In the Spark context try using: SparkContext.addPyFile(“module.py”) # Also supports .zip Quoting from the docs: Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or … Read more
Use Rtree (examples) as R-tree index to: (1) index the bounds of the 36k polygons (do this just after reading jsonfile), then (2) very quickly find the intersecting bounding boxes of each polygon to your point of interest. Then, (3) for the intersecting bounding boxes from Rtree, use shapely to use, e.g. point.within(p) to do … Read more
Download WHL file using http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely Install: python -m pip install Shapely-1.5.17-cp36-cp36m-win_amd64.whl Ubuntu: pip install shapely==1.6b2
For questions 2-4, you have to have GEOS installed on your system. If you have homebrew you can do the following: brew install geos Install homebrew here if you don’t have it: Link
The GIS term you are describing is linear referencing, and Shapely has these methods. # Length along line that is closest to the point print(line.project(p)) # Now combine with interpolated point on line p2 = line.interpolate(line.project(p)) print(p2) # POINT (5 7) An alternative method is to use nearest_points: from shapely.ops import nearest_points p2 = nearest_points(line, … Read more
Installed shapely using pip, and had the same problem. So I went ahead and installed it like so: sudo apt-get install libgeos-dev And it worked. I’m running Ubuntu, so if you’re on Fedora, you should run: sudo yum install geos-devel On MACOS you can run: brew install geos
Consider using Rtree to help identify which grid cells that a polygon may intersect. This way, you can remove the for loop used with the array of lat/lons, which is probably the slow part. Structure your code something like this: from shapely.ops import cascaded_union from rtree import index idx = index.Index() # Populate R-tree index … Read more