spark ssc.textFileStream is not streamining any files from directory
Try it with another directory and then copy these files to that directory, while the job is running.
Try it with another directory and then copy these files to that directory, while the job is running.
Most filesystems (but not all) use locking to guard concurrent access to the same file. The lock can be exclusive, so the first user to get the lock gets access – subsequent users get a “access denied” error. In your example scenario, user A will be able to read the file and gets the file … Read more
What you’re doing seems perfectly OK, provided you’re using the POSIX “raw” IO syscalls such as read(), write(), lseek() and so forth. If you use C stdio (fread(), fwrite() and friends) or some other language runtime library which has its own userspace buffering, then the answer by “Tilo” is relevant, in that due to the … Read more
From: http://www.grouplogic.com/knowledge/index.cfm/fuseaction/view_Info/docID/111. The following characters are invalid as file or folder names on Windows using NTFS: / ? < > \ : * | ” and any character you can type with the Ctrl key. In addition to the above illegal characters the caret ^ is also not permitted under Windows Operating Systems using the … Read more
FAT32: Maximum number of files: 268,173,300 Maximum number of files per directory: 216 – 1 (65,535) Maximum file size: 2 GiB – 1 without LFS, 4 GiB – 1 with NTFS: Maximum number of files: 232 – 1 (4,294,967,295) Maximum file size Implementation: 244 – 26 bytes (16 TiB – 64 KiB) Theoretical: 264 – 26 bytes (16 EiB – 64 KiB) Maximum volume size Implementation: 232 – 1 clusters (256 TiB – 64 KiB) Theoretical: 264 – 1 clusters (1 YiB – 64 KiB) ext2: Maximum number of files: 1018 … Read more