Low priority, but currently the Blake3 code will not compile on x64 systems without the AVX2 / AVX512 instructions, at least with the following test system.

Test system
System: MacPro 5,1
CPU: Xeon Westmere W3680
OS: macOS 10.14

Compile errors

/Users/xxxx/Documents/tmp/chiapos/src/b3/blake3_avx2.c:20:42: error: always_inline function '_mm256_set1_epi32' requires target feature 'xsave', but would be inlined into function 'set1' that is
      compiled without support for 'xsave'
INLINE __m256i set1(uint32_t x) { return _mm256_set1_epi32((int32_t)x); }

This above is repeated for all the set, and, cmpgt, xor etc functions and also in the 512 files:

/Users/xxxx/Documents/tmp/chiapos/src/b3/blake3_avx512.c:1115:3: error: always_inline function '_mm256_mask_storeu_epi32' requires target feature 'sse4.2', but would be inlined into function
      'blake3_hash16_avx512' that is compiled without support for 'sse4.2'
  _mm256_mask_storeu_epi32(&out[0 * sizeof(__m256i)], (__mmask8)-1, _mm512_castsi512_si256(padded[0]));

I've confirmed that the get_cpu_features() function is currently identifying that AVX is not available:

SSE2: 1
SSE3: 2
SSE4.1: 4
AVX: 0
AVX2: 0
AVX512F: 0
AVX512VL: 0

However the compile itself is still including the instructions that are not available.

Workaround
The build can be made to work by removing the src/b3/blake3_avx2.c and src/b3/blake3_avx512.c files from BLAKE3_SRC in CMakeLists.txt and then:

#define BLAKE3_NO_AVX512
#define BLAKE3_NO_AVX2

Not necessarily expecting a fix on this, but it looked like there had been some work into trying to do CPU feature identification so wanted to call it out.

I noticed that the temp files are very fragmented on my disks causing a larger amount of seeks than necessary.

I created a proof of concept version of chiapos where it pre allocates files, causing them to stay mostly continuous. Code is mostly copied from Windows C++ API documentation and stackoverflow as I am not familiar with C++.

On one of my mechanical disks this already cut down phase 1 time by ~10% and another disk will soon be starting phase 2 as well.

I cannot tell what effect this will have on the other phases yet.

Pre allocation is limited to 2GB above file size with the method I used, but this could easily be worked around by pre allocating more space every 2GB written, which would already improve fragmentation a lot. This limitation is Windows only.

Note that over allocating a file above its final size does not corrupt it. It gets truncated to its final size once the handle is closed, so not knowing the exact final size is not a problem.

static LARGE_INTEGER toLargeInteger(long long value)
{
    LARGE_INTEGER result;

#ifdef INT64_MAX // Does the compiler natively support 64-bit integers?
    result.QuadPart = value;
#else
    result.high_part = value & 0xFFFFFFFF00000000;
    result.low_part = value & 0xFFFFFFFF;
#endif
    return result;
}

bool hasEnding(std::string const& fullString, std::string const& ending)
{
    if (fullString.length() >= ending.length()) {
        return (0 == fullString.compare(fullString.length() - ending.length(), ending.length(), ending));
    }
    else {
        return false;
    }
}

void allocate(FILE* file, std::string filename)
{
    HANDLE hFile = (HANDLE)_get_osfhandle(_fileno(file));
    if (hFile != INVALID_HANDLE_VALUE) {
        FILE_ALLOCATION_INFO fai;
        std::cout << "filename is " << filename << std::endl;
        if (hasEnding(filename, "plot.tmp") || hasEnding(filename, "plot.2.tmp") || hasEnding(filename, "plot")) {
            // apparently there is a limit of 2 GB
            std::cout << "allocating <2GB" << std::endl;
            fai.AllocationSize = toLargeInteger(2 * 1000L * 1024L * 1024L); // plot.tmp
        }
        else if (hasEnding(filename, "plot.table1.tmp") || hasEnding(filename, "plot.table2.tmp") || hasEnding(filename, "plot.table3.tmp") || hasEnding(filename, "plot.table4.tmp") || hasEnding(filename, "plot.table5.tmp") || hasEnding(filename, "plot.table6.tmp") || hasEnding(filename, "plot.table7.tmp")) {
            std::cout << "allocating <2GB" << std::endl;
            fai.AllocationSize = toLargeInteger(2 * 1000L * 1024L * 1024L); // plot.table7.tmp
        }
        else if (hasEnding(filename, "sort.tmp")) {
            std::cout << "allocating 400MB" << std::endl;
            fai.AllocationSize = toLargeInteger(400L * 1024L * 1024L); // sort.tmp
        }
        else {
            std::cout << "allocating 400MB" << std::endl;
            fai.AllocationSize = toLargeInteger(400L * 1024L * 1024L); // sort_bucket_084.tmp
        }

        BOOL fResult = SetFileInformationByHandle(hFile,
            FileAllocationInfo,
            &fai,
            sizeof(FILE_ALLOCATION_INFO));

        if (fResult) {
            std::cout << "allocation worked" << std::endl;
        }
        else {
            // error code 87 appears when allocation size more than 2GB above file size. No other error appeared so far
            std::cout << "allocation error: '" << GetLastError() << "' continuing without" << std::endl;
        }
    }
    else {
        throw std::invalid_argument("invalid file handle (_get_osfhandle _fileno failed?)");
    }
}

[...]

void Open(uint8_t flags = 0)
{
    // if the file is already open, don't do anything
    if (f_)
        return;

    // Opens the file for reading and writing
    do {
#ifdef _WIN32
        f_ = ::_wfopen(filename_.c_str(), (flags & writeFlag) ? L"w+b" : L"r+b");
        if ((flags & writeFlag)) {
            allocate(f_, filename_.string());
        }
#else
        f_ = ::fopen(filename_.c_str(), (flags & writeFlag) ? "w+b" : "r+b");
#endif
[...]

Tested on current Windows 10, Seagate Barracuda 2TB (2016) Disks, NTFS with cluster size of 4096 Bytes.

In latest Blake3 project you'll find the optimizations for the extensions SSE2 & Neon.
https://github.com/BLAKE3-team/BLAKE3/tree/master/c
Could you update your dependency to include latest Blake3 project?

Describe the bug
every second day system shows the error message and farming stops

Expected behavior

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu Linux
• OS Version/Flavor: 20.04
• CPU: Ryzen 7 3700X

Additional context
Add any other context about the problem here.

Hi,

this could be hw related, but however, maybe software can/should get out of the loop and print some error message, so one can easier investigate what the problem is. And also, stop waisting resources, rather terminate if can't make any progress.

At our side it stops at random places, in the 3rd phase mostly. I have all the data ( p3.t2.sort_bucket_xxx.tmp ) saved at the point it stopped, if you want me to enable some logging/debugging from here on. Last message I had was about sorting bucket 53, in table 1/2 in phase 3.

I took a look into the code very briefly, and I can see several while-s going on here, so some of them might not be terminating properly. Please note - no disk activity is happening, while the processor thread is running at 100%.

If you suspect hardware, what it would be? RAM / DISK or something else?

If you give me some instructions how to run more verbose/debugging version, I can do that and maybe that will nail down this problem for me and for others.

This seems related:
Chia-Network/chia-blockchain#3228 (comment)

On windows with a drive mountpoint at: C:_\SM01\05 - plotting to this directory path fails with a Temp dir not found. If I create a subdir below it, eg C:_\SM01\05\PLOT - plot works properly.

Log file attached.

Describe the bug
Sometimes I find one my harvesters not talking to the farmer anymore.
This happened now several times on different machines in the last couple of days. Including Ubunut Server, Ubuntu Desktop and a raspberry pie with raspbian.
I updated to chia 1.0.5 on all devieces over the last couple of days.

To Reproduce
Steps to reproduce the behavior:
Run the harvester and wait couple of days/hours until it stops connecting to the farmer

Expected behavior
The harvester to continue harvesting :)
When such an error occurs, it should at least restart itself.
Will probably write a script in the short term that restarts the harvester regularly.

Screenshots
07:19:29.127 daemon chia.daemon.server : INFO Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
message = await self.read_message()
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
frame = await self.read_frame(max_size)
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
frame = await Frame.read(
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
data = await reader(2)
File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/sebastian/chia-blockchain/chia/daemon/server.py", line 165, in safe_handle
async for message in websocket:
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in aiter
yield await self.recv()
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
await self.ensure_open()
File "/home/sebastian/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

Desktop (please complete the following information):

• OS: Ubuntu Desktop, Ubuntu Server, Raspbian
• OS Version/Flavor: Ubuntu 20.04., Raspbian 64bit
• CPU: Different

Additional context
I can retrieve more info from the different machines, but as it seems to appear radomly everywhere, might be a genearal thing?
Happens with LAN connection at home, data center LAN and WiFi at home

Analyzing your Plotting Application in steps 2/3: It is bound by memory copy and seek speed - the abstraction you chose for a harddisk in disk.hpp is a streaming device.

Much better - use memory mapped IO directly. No buffer needed at all. The operating system maps its IO pages directly into your application. A 100GB file becomes a pointer where you can read and write in memory - but in reality you are writing to disk in a very smart way (details are handled by the OS). This is much faster than random seeking - even with the buffers you are using.

Win32: CreatefileMappingW
Linux: mmap

A quick and dirty win32 CreatefileMappingW impl for disk.hpp runs 40% faster for step 2 on my machine
1588s -> 981s

This is still with copying all the bytes from pointer to pointer.

With memory mapped IO there is no need to copy anything!

The fastest way to find in a file (uses the same idea) is here: https://stackoverflow.com/questions/17925051/fast-textfile-reading-in-c
(Only works for non resizable files (sorting is PERFECT))

I put chiapos1.0.1 into chia-blockchian, but when I execute ‘python -m pip install ./chiapos-1.0.1’, an error occurs:

LookupError: setuptools-scm was unable to detect version for '/opt/data/tmpf/pip-req-build-_2t868_b'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

This seems to be unable to find the .git folder, I want him to use the .git folder in the upper directory

Describe the bug
Some of my plots (say 25%) freeze in phase 2, i.e. no further progress is reported after the freeze, for hours (then I kill the process manually).
The log files do not continue. No new files in the temp dir are created.
Otherwise, the process seems to continue: CPU time is being used similar to alive plots, ram stays normal. File locks on log file still exist.

Normally a plot finishes in 9 hours, I usually kill the stuck ones after 12 or so.

Plots were started with

chia plots create -k 32 -r 4 -u 128 -b 4520 -t /mnt/chiatmp -d /mnt/chiaplots -e

To Reproduce
Don't know. For me it's just a chance of say 25%.

Expected behavior
Plot should not freeze

Screenshots
Not applicable.

Desktop (please complete the following information):

• OS: Linux
• OS Ubuntu 20.04 LTS latest updates
• CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz

Additional context

• Chia beta 27, freshly installed as instructed (no update)
• Temp dir is formatted as ext4. It's a Corsair MP600 NVME SSD.
• Always more than 25gb ram free, swap disabled
• Temp dir always has more than 300gb free

Please ask for further info. I will attach log files of three such occurences.

Describe the bug
When creating plot number 2 the log reports a repeating error, and never finishes:

Only read 0 of 4194304 bytes at offset 51585548112 from "E:\temp\plot-k32-2021-02-22-17-54-5cb69645d57826c3c49e7a6461025002ec617399e5dd234286096b0ffd588d38.plot.table7.tmp"with length 51598975152. Error 1. Retrying in five minutes._

this keeps repeating

Steps to reproduce the behavior:
Create a new plot.
Wait for phase 2/4 to begin

Expected behavior
The plot should finish, it doesnt.

Desktop (please complete the following information):

• OS: Win10 pro
• OS Version/Flavor: 10.0.19041 build 19041
• CPU: i5-9400F
• Chia version: 0.1.27

When plotting stops due to no space left on the device and then resumes, it doesn't say so. It's not a big problem during the plotting because the process is verbose enough to get some message after it resumes.
But when this happens during the move of the final plot to -d, the user will not be notified about the resume of this copy operation until it's finished.

Scanning dependencies of target RunTests
[ 30%] Building CXX object CMakeFiles/RunTests.dir/tests/test-main.cpp.o
In file included from /usr/include/signal.h:315,
from /home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/../lib/include/catch.hpp:8034,
from /home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/test-main.cpp:16:
/home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/../lib/include/catch.hpp:10822:58: error: call to non-‘constexpr’ function ‘long int sysconf(int)’
10822 | static constexpr std::size_t sigStackSize = 32768 >= MINSIGSTKSZ ? 32768 : MINSIGSTKSZ;
| ^~~~~~~~~~~
In file included from /usr/include/bits/sigstksz.h:24,
from /usr/include/signal.h:315,
from /home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/../lib/include/catch.hpp:8034,
from /home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/test-main.cpp:16:
/usr/include/unistd.h:630:17: note: ‘long int sysconf(int)’ declared here
630 | extern long int sysconf (int __name) __THROW;
| ^~~~~~~
In file included from /home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/test-main.cpp:16:
/home/ss/rpmbuild/BUILD/chiapos-1.0.2/tests/../lib/include/catch.hpp:10881:45: error: size of array ‘altStackMem’ is not an integral constant-expression
10881 | char FatalConditionHandler::altStackMem[sigStackSize] = {};
| ^~~~~~~~~~~~

Describe the bug
Remote harvester daemon server segfault with a Websocket exception. The daemon stops running the harvester. Required steps to restart:

chia stop all -d
chia start harvester

Log

10:19:52.926 daemon src.daemon.server          : INFO     Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 846, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia/Projects/chia-blockchain/src/daemon/server.py", line 149, in safe_handle
    async for message in websocket:
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/chia/Projects/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

dmesg

[388966.187137] chia_harvester[39153]: segfault at 2 ip 00007f225b187d01 sp 00007f221effb988 error 4 in chiapos.cpython-38-x86_64-linux-gnu.so[7f225b11b000+b8000]
[388966.187156] Code: 39 c6 75 e5 31 c0 c3 66 90 c3 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 41 57 48 01 fe 41 56 48 8d 46 fd 41 55 41 54 55 53 <66> 41 83 78 02 00 48 89 44 24 b8 48 c7 c0 fd ff ff ff 48 89 7c 24

Expected behavior
Remote harvester continues running.

Server

• OS: Linux
• OS Version/Flavor: Ubuntu 20.04.2 LTS
• CPU: Intel 8Cx3.2GHz (Silver 4215R)
• Chia version: beta-1.0b27

CMake Error at CMakeLists.txt:158 (target_compile_features):
target_compile_features The compiler feature "cxx_std_17" is not known to
CXX compiler

"GNU"

version 4.8.5.

Description

There appears to be an issue regarding how the plot is moved to the final directory. In the logs below, it appears that the condition tmp_2_filename.parent_path() == final_filename.parent_path() evaluated to false (link to code). This led to almost 1h copying a file that could simply have been moved.

My understanding is that this works across operating systems (specifically interop between Windows and *nix based OSes).

Current behavior

Final file copied from /home/[redacted]/tmp to /home/[redacted]/plots.

Expected behavior

Unclear if this needs per OS code but I would suspect that in *nix based OSes, we could always move the file (maybe not, I'm unsure). The higher effort alternative would be to understand if the different paths belong to the same mount point/device.
For Windows, my understand is that we could compare the first token of fs::path (not certain) to see if the drive letters match.

Logs:

lot-hdd.log-Copied final file from "/home/[redacted]/tmp/plot-k32-2021-04-27-04-24-[redacted].plot.2.tmp" to "/home/[redacted]/plots/plot-k32-2021-04-27-04-24-[redacted].plot.2.tmp"
plot-hdd.log-Copy time = 3150.589 seconds. CPU (7.240%) Tue Apr 27 17:23:05 2021
plot-hdd.log-Removed temp2 file "/home/[redacted]/tmp/plot-k32-2021-04-27-04-24-[redacted].plot.2.tmp"? 1
plot-hdd.log:Renamed final file from "/home/[redacted]/plots/plot-k32-2021-04-27-04-24-[redacted].plot.2.tmp" to "/home/pedro/plots/plot-k32-2021-04-27-04-24-[redacted].plot"

Hi i have proble of sync i reboot my pc 4.22 a.m and now I wait few hours to sync and nothing do

Suggested enhancement:

Make the console / log output more verbose, or at least announce the start of this sometimes time consuming action.

What will it solve:

Premature termination of copying progress due to assuming the plotter crashed.

Why is it needed:

As a beginner myself I was confused that after the barrage of logs that is the plotting, and seemingly it saying it is finished by reporting total times. The script went silent and just sat there. The GUI reports it is still plotting, yet the console clearly says it is done.
This makes it probable for a beginner to think (like I did) that the plotter crashed, quit the progress, and does the move / cleanup themselves. This can result in corrupted plots, and it has crashed windows explorer (do not rename target files when this script is running folks).

Current situation:

The log outputs as follows:

text in [...] are edits to keep the example readable.

Approximate working space used (without final file): 286.550 GiB
Final File size: 101.311 GiB
Total time = [Time] seconds. CPU (66.530%) [Date]
Copied final file from "D:\\tmp\\plot-k32-[ID].plot.2.tmp" to "F:\\plot-k32-[ID].plot.2.tmp"
Removed temp2 file "D:\\tmp\\[ID].plot.2.tmp"? 1
Renamed final file from "F:\\[ID].plot.2.tmp" to "F:\\[ID].plot"

Enhanced situation:

The enhancement can 'simply'-I know it never always is- be implemented as follows:

Text in [...] are edits to keep the example readable.
Text after > are to mark new lines.

Approximate working space used (without final file): 286.550 GiB
Final File size: 101.311 GiB
Total time = [Time] seconds. CPU (66.530%) [Date]
> Start of copying final file from "D:\\tmp\\plot-k32-[ID].plot.2.tmp" to "F:\\plot-k32-[ID].plot.2.tmp". This might take a while.
Copied final file from "D:\\tmp\\plot-k32-[ID].plot.2.tmp" to "F:\\plot-k32-[ID].plot.2.tmp"
Removed temp2 file "D:\\tmp\\[ID].plot.2.tmp"? 1
Renamed final file from "F:\\[ID].plot.2.tmp" to "F:\\[ID].plot"

Or to satisfy optimizers, you might add a periodic write/copy speed report or as part of the Copied final file-line.

How to reproduce:

1. Unplug HDD during compression step
2. Plotter detects incomplete write and enters retry loop (every 5 minutes)
3. Reconnect HDD -> plotter won't be able to successfully resume

Seems to me that the file needs to be reopened inside the loop. Or perhaps this use-case was not intended to be a feature?

Plotting fails to start when plotting to an NTFS Mountpoint via SMB share.

Remote Windows box:
MP (NTFS Mount point and SMB share)

• Subdir (Subdirectory of MP )
• MP2 (Subdirectory and also a NTFS Mountpoint)

Plotter:
SMB share to \\x.x.x.x\MP

From both Windows and Linux plotting using the root of the share fails:

ID: c515b361ca2b44977b97dd343930c2cde19c1d672bfbcd3208769298b830e093
Plot size is: 32
Buffer size is: 4508MiB
Using 128 buckets
Using 1 threads of stripe size 65536
Caught plotting error: Could not open /home/haitch/temp2/plot-k32-2020-11-01-08-58-c515b361ca2b44977b97dd343930c2cde19c1d672bfbcd3208769298b830e093.plot.sort.tmp: Permission denied
Traceback (most recent call last):
  File "/home/haitch/chia-blockchain/venv/bin/chia", line 33, in <module>
    sys.exit(load_entry_point('chia-blockchain', 'console_scripts', 'chia')())
  File "/home/haitch/chia-blockchain/src/cmds/chia.py", line 59, in main
    return chia(args, parser)
  File "/home/haitch/chia-blockchain/src/cmds/chia.py", line 53, in chia
    return args.function(args, parser)
  File "/home/haitch/chia-blockchain/src/cmds/plots.py", line 146, in handler
    create_plots(args, root_path)
  File "/home/haitch/chia-blockchain/src/plotting/create_plots.py", line 137, in create_plots
    plotter.create_plot_disk(
RuntimeError: std::exception

On both Windows and Linux, plotting to \\x.x.x.x\MP\Subdir does not throw an error.

On Linux, plotting to \\x.x.x.x\MP\MP2 does not throw an error.

On Windows only - plotting to \\x.x.x.x\MP\MP2, plotting again fails.

ID: 0c4af82786302bfe833e138e1e70938d394bdb2a33af13eef3034d99b7639f7f
Plot size is: 33
Buffer size is: 4600MiB
Using 128 buckets
Using 1 threads of stripe size 65536
Caught plotting error: Temp directory z:\chia11 does not exist
Traceback (most recent call last):
  File "src\cmds\chia.py", line 63, in <module>
  File "src\cmds\chia.py", line 59, in main
  File "src\cmds\chia.py", line 53, in chia
  File "src\cmds\plots.py", line 146, in handler
  File "src\plotting\create_plots.py", line 148, in create_plots
RuntimeError: Unknown exception
[6896] Failed to execute script chia

Reported by @sadams25 in Chia-Network/chia-blockchain#263.

If your machine does not have enough memory, the windows plotter fails silently in the UI and no log is created. I had to discover the issue by running the CLU. That being said, this is a pretty minor issue as I was running this on a very underpowered VM, which is my fault.

The Fault:

There is a kind of plot corruption where the corruption is only found after m proofs, in order for "plots check" to find this
corruption n needs to be at least m. Due to the nature of the proof finding the chance to find the corruption is very small and increases with the distance between n and m.

The Problem:

There is no way to reliably identify plots with this kind of corruption due to a k32 plot containing proofs in the realm of 4296495116 (if I understand correctly) which would not be feasibly to check for in reasonably time for any amount of plots.

Free text:

I have a plot that is only detected as faulty if the number of found proofs is greater or equal 14.
The output of the checks can be found here.
Before I also had a plot that had this problem if the number of proofs reached 33, but I have already deleted that one.
I tried with n>30 multiple times and every time the error occurred. With n = 25 it occasionally occurred.

Keep this in mind: pypa/pyproject-hooks#91

Xorinox plotted with a tmp2 directory that did not have enough space. The plotting finished with no error message, but the final plot was only 3GB for a k=30, so it did not succeed.

I am having problem with the phase 3, it process every bucket 2 times with half ram... when the pc has 50% free ram.

Bucket 29 uniform sort. Ram: 1.906GiB, u_sort min: 1.250GiB, qs min: 0.328GiB.
Bucket 29 uniform sort. Ram: 1.906GiB, u_sort min: 0.500GiB, qs min: 0.250GiB.
Bucket 30 uniform sort. Ram: 1.906GiB, u_sort min: 1.250GiB, qs min: 0.329GiB.
Bucket 30 uniform sort. Ram: 1.906GiB, u_sort min: 0.500GiB, qs min: 0.250GiB.
Bucket 31 uniform sort. Ram: 1.906GiB, u_sort min: 1.250GiB, qs min: 0.329GiB.
Bucket 31 uniform sort. Ram: 1.906GiB, u_sort min: 0.500GiB, qs min: 0.250GiB.
Bucket 32 uniform sort. Ram: 1.906GiB, u_sort min: 1.250GiB, qs min: 0.329GiB.
Bucket 32 uniform sort. Ram: 1.906GiB, u_sort min: 0.500GiB, qs min: 0.250GiB.

I have 32gb, and currently have 5 process with 4000 MB each.

The process monitor reports 44% of ram in use... what can i do? im testing but i think that the plot time is going up..

Describe the bug
I believe the issue occurs when the harvester aims to read proof data while the OS is umounting a USB3 drive.

Expected behavior
Harvester moves on after logging to debug.log that there was a critical read error, without the need to be restarted.

Additional context
beta27

08:14:42.243 harvester src.harvester.harvester : ERROR    Error using prover object. Reinitializing prover object. Caught an unknown exception!
08:14:42.281 harvester src.harvester.harvester : ERROR    Error reinitializing plot /chia/plots/disk04/plot-k32-2021-01-10-23-25-c147d7110d74f4d4655e62ef7f2c320c4bc36236c5d6efccce55d7cd75c60bdc.plot. Caught an unknown exception!
08:14:42.527 daemon src.daemon.server          : INFO     Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/chia-blockchain/src/daemon/server.py", line 149, in safe_handle
    async for message in websocket:
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/opt/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

Describe the bug
When plotting is in progress, the progress bar does not change, is empty and silver all the time.

I can View Log for selected plot, but I do not understand the individual steps, so, if I see "Computing table 4" and "Bucket 42 uniform sort" after 3 days of running, I can't tell if it makes sense to let this activity run on this computer. Will it run for another hour or another year? Progress bar showing still 0. A rough estimate, or at least some progress on the progress bar, would suffice.

To Reproduce
Steps to reproduce the behavior:

1. Open Chia Blockchain UI
2. On Plots, add a plot and see progress in column Filename

Expected behavior
Progress bar showing progress in plotting, or I can in any way find out the estimated time of the plotting.

Desktop (please complete the following information):

• OS: Windows
• OS Version/Flavor: Windows 10 20H2
• CPU: Intel(R) Core(TM) i7

Additional context
GUI 0.1.27

Describe the bug
Plotting fails in phase 2, when compressing tables 2 and 3. The plot reports "Not enough memory for sort in memory".
Configuration: k = 32; swap space available = 512 GB; target space available = 8 TB; memory allocated = 8 GB.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Plots'.
2. Click on 'New plot'.
3. Select 'k = 32'.
4. In 'Advanced Configuration', allocate 8 GB of memory; leave the number of buckets to 0.
5. Click on 'Open log'.
6. Error shows up when compressing tables 2 and 3

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: Windows 10
• CPU: AMD Ryzen 7 5800X

Additional context
This behaviour seems identical to what reported on Reddit for Beta 18. The post suggested downgrading to Beta 17 until Beta 19 was available. I assumed Beta 19 fixed that bug, so I upgraded straight from 17 to 19, but the bug still seems to be there. I am now plotting by forcing 64 buckets. I will report if that works!

I managed to create a bad plot that segfaults sometimes when getting qualities
It's a k18 though, so this issue might not be present in the big ones
anyway, it might be worth exploring. The plot id is: "22d67d64197d6787da61cd31302832ab8667097239e38cf442eb8a5b19cbc074"

Made using stripe size of 2000, b=100.

With a "chia plots check" of 200,000, found a crash:
munmap_chunk(): invalid pointer

harvester can not connected
xxx@xxxxxxxxx:/media/plot/ST_4T_01/farm2$ chia farm summary
Connection error. Check if harvester is running at 8560
Farming status: Farming
Total chia farmed: 0.0
User transaction fees: 0.0
Block rewards: 0.0
Last height farmed: 0
Plot count: Unknown
Total size of plots: Unknown
Estimated network space: 5020.860 PiB
Expected time to win: Unknown
Note: log into your key using 'chia wallet show' to see rewards for each key

in syslog
May 15 01:25:41 xxxxxxxxxxxxx kernel: [383060.769999] chia_harvester[122631]: segfault at 7f3970113bbb ip 00007f39b16e31e2 sp 00007f39acc868f8 error 4 in chiapos.cpython-38-x86_64-linux-gnu.so[7f39b1676000+b8000]
May 15 01:25:41 xxxxxxxxxxxxx kernel: [383060.770011] Code: d9 48 8b 7c 24 c8 49 89 d3 0f bd c0 83 f0 1f 8d 74 c8 29 e9 fb fc ff ff 48 85 c9 0f 84 5a ff ff ff 48 8d 5a 08 48 89 54 24 e0 <0f> b6 74 0a ff 48 89 5c 24 e8 48 83 f9 07 0f 86 5a 02 00 00 4c 8d

package: chia-blockchain_1.1.5_amd64.deb
os: ubuntu x86-64 20.04

Why is the final file (*.plot.2.tmp) in Phase 4 written to the tempdir and then copied and renamed to the final destination directory instead of writing it there in the first place?
This could save us ~30 Minutes!

Users can still select Plot size of K=25 ... which does not qualify for farming on the mainnet

K=32 is the min.

I suppose for testing maybe but you may want to add some disclaimer when this is chosen

Describe the bug

When the plotter reaches table 6 for backpropagation the process randomly exits. The last message just says 'Killed', which is not found in the log and only visible if run trough CLI.
Setting the buffer size to anything lower/higher didn't make any difference.
Plotting with k=25 works just fine.
Also, I've run the process 3 times now (with different ram settings) and it ALWAYS crashed at backpropagation on table 6.

Sample Output

...
Starting phase 2/4: Backpropagation into tmp files... Mon Apr 19 15:17:10 2021
Backpropagating on table 7
scanned table 7
scanned time =  194.997 seconds. CPU (45.350%) Mon Apr 19 15:20:25 2021
sorting table 7
Backpropagating on table 6
Killed
(venv) chia@chia:~/chia-blockchain$

Log File: plotter_log_43b3a6a3-c5cd-47a8-aa94-dbbe82342a85.txt

Environment

OS: Debian 10.9 64bit (4.19.0-16-amd64)
CPU: Intel Xeon E5-2420 v2
RAM: 32gb ECC
Temp/Final Disk: 16TB Exos X16 (one drive for both)
File System: ext4

Most of the build seems to run and near the end I see the following:

[ 88%] Building ASM object CMakeFiles/ProofOfSpace.dir/src/b3/blake3_sse41_x86-64_unix.S.o
[ 91%] Building ASM object CMakeFiles/chiapos.dir/src/b3/blake3_sse41_x86-64_unix.S.o
gmake[2]: *** [CMakeFiles/ProofOfSpace.dir/build.make:82: CMakeFiles/ProofOfSpace.dir/src/cli.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:150: CMakeFiles/ProofOfSpace.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake[2]: *** [CMakeFiles/RunTests.dir/build.make:82: CMakeFiles/RunTests.dir/tests/test-main.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:123: CMakeFiles/RunTests.dir/all] Error 2
[ 94%] Linking CXX shared module ../lib.linux-x86_64-3.8/chiapos.cpython-38-x86_64-linux-gnu.so
[ 94%] Built target chiapos
gmake: *** [Makefile:114: all] Error 2
Traceback (most recent call last):
File "setup.py", line 203, in
setup(
File "/usr/lib/python3.8/site-packages/setuptools/init.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib64/python3.8/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib64/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "setup.py", line 38, in run
self.build_extension(ext)
File "setup.py", line 70, in build_extension
subprocess.check_call(
File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j', '6']' returned non-zero exit status 2.
error: Bad exit status from /var/tmp/rpm-tmp.rhA1XY (%build)

Any thoughts or anything you need me to provide? I am quite sure I meet all the dependencies.

Hello,

I have a i7 3770k which is and old CPU with 8go Ram

I have tried to create the plot accross the GUI or script but both always fail with the 2nd plot creation

I've made a simple script that will format the Temp SSD drive before creating the 2nd plot but this didnt help at all

function plot{
if ($i -eq 0){
Write-host "Work Done"
}else{
$i
Format-Volume -DriveLetter G -FileSystem NTFS -NewFileSystemLabel CHIA_SSD
cd $env:USERPROFILE\AppData\Local\Chia-Blockchain\app*\resources\app.asar.unpacked\daemon
.\chia.exe version
.\chia.exe plots create -k 32 -n 1 -t "G:" -d "H:" -b 3389 -u 128 -r 2
$i--
plot
}
}
$i=6 #set the nb of plot you want to create
plot

You can see my plot settings in the scripts :
.\chia.exe plots create -k 32 -n 1 -t "G:" -d "H:" -b 3389 -u 128 -r 2

The error is every time the same :
Only wrote * bytes at * with length * . Error 1. Retrying in five minutes.

I'm running Windows 10 and my FileSystem is NTFS, Memtest tell me everything is ok.

What should I do ?

I can continue to grow my farm doing 1 plot after 1, but it's not really powerfull :/

Thanks for reading

ADD: I see that k32 need 239GiB and my SSD space is 238Go
I have not enough space then ? Cause > 239 GiB = 256.624295936 GB
But why can I create 1 plot, but not 2... I'm lost

It would be very useful to be able to restart a crashed or stopped plot from the end of a completed stage which has the tmp files available. Have had a couple of crashes where I've managed to lose hours of plotting because I've had to restart from the beginning. Unlikely IMO but could also be used to distributed different phases across different machines?

Perhaps -ph, --phase ?

Final Directory not printed in running plotter logfiles, only printed at the end.

I was asked to show the Final Directory in ChiaPlotStatus. The problem is the final directory is logged at the end of phase 4, so I cannot show the final directory of running plotting processes.

Pull Request incoming.

,

There is one thing I would love to see added to the plotting logs: the time it took to transfer from -t to -2, and from -2 to -d.

I've had terrible stats there:

• Transfer on a USB 2 connection
• Transfer on a slow NFS connection

Having those transfer times in the logs would help identifying that potential bottleneck :)

Chia Blockchain 1.1.5 affect the plot of chiapos

Support for farming uncompressed plots

Proposal

I would like to propose a new feature/enhancement for farming uncompressed plots along with a new file extension for these uncompressed plots (something along the lines of a .uplot) to ensure compatibility.

Background

The generation of tables (especially larger K values) take time, particularly for those joining with limited compute hardware, but the temporary files generated during the plotting process should contain the same data as the compressed final products. Implementing a method for farming from these uncompressed temporary files would allow lower power devices to begin harvesting from these plots sooner. The farmer could then optionally return at a later date to optimize their plots by converting them to the compressed final form.

Example use case

New farmer joins with low power hardware, but lots of space (say a low to mid range NAS device that was previously under utilized). They would be able to generate uncompressed plots (I believe this might reduce compute time by up to 50% as compression takes place in phase 3 and phase 4 being mostly 'house work'). This would require that they dedicate about 3x the amount of storage used by a compressed .plot (maybe this could be called a .plotx or .xplot) until they comeback and compress at a later time (say once they have reach >75% utilization of their existing storage space or purchased more powerful compute hardware).

我在linux 下使用 chiapos 创建K32 plot 文件。
910GiB的硬盘 同时开始了3个任务。结果提示磁盘已满。
然后测试了2个任务同时运行。发现，最大占用超过了620GiB

运行参数：-r 4 -u 128 -s 131072 -t /pcache1 -2 /pcache1 -d /pdisk1 -e false -b 8000

请问这是BUG 吗。还是我的参数有问题。。。或者说 必须在WINDOWS系统下才是只用 239G？

Hello, i compile plotter for windows, work fine, but if i check plot via original chia.exe:
2021-05-15T05:58:03.707 chia.plotting.plot_tools : ERROR Failed to open file E:\test\plot.plot. Invalid number of bytes 31 Traceback (most recent call last). What can i do? How i can use that plots compiled by my own plotter for original chia gui, harvester etc. Thanks!

Background

When plotting, it is not readily apparent what "2nd temporary directory" (I'll refer to it as tmp2/tmp2_dirname here) is used for. Having looked into the code, I noticed that tmp2 is only used starting in phase 3, when the plotter compresses intermediate data from phase 2 and writes it to tmp2. This behavior isn't well documented nor expected. The description in the GUI may mislead users into thinking tmp2 is used as an overflow directory.

My attempt to plot using a 240GB NVMe SSD I had lying around gets stuck in phase 2, when FileDisk.Write() fails to write the entire file. The plotter spins infinitely in the do ... while loop of the Write() method.

Proposal

utilize tmp2/tmp2_dirname in all phases as needed. In essence, implement functionality that would effectively use tmp2 as an overflow if FileDisk.Write() fails due to hitting disk space limit (or for any other reason), in phase 1 & 2.

Possible solution

DiskPlotter.CreatePlotDisk() constructs vector<FileDisk> tmp_1_filenames based on tmp_dirname only. It may be feasible to:

• construct vector<FileDisk> tmp_2_overflow_filenames using the same logic, but based on tmp2_dirname. This vector should be of the same size
• when calling (b17)RunPhaseX(), pass both tmp_1_filenames and tmp_2_overflow_filenames, possibly along with
• a simple data structure to track which filename was used/written

this is a mamagical code!
can the plotter be accelerated by cuda or rocm or opencl?

Description

On Ubuntu 20.04 I cloned chiapos and followed the compile instructions.

I see the following error message.

$ cmake ../
-- pybind11 v2.6.2
-- cxxopts version 2.2.0
-- Configuring done
CMake Error in CMakeLists.txt:
  Imported target "pybind11::module" includes non-existent path

    "/usr/include/python3.8"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and references files it does not
  provide.

I tried the following but the error remains.

$ git clone https://github.com/Chia-Network/chiapos.git
$ cd chiapos
$ python3 -m venv venv
$ . venv/bin/activate
$ python --version
Python 3.8.5
$ pip install -U pip setuptools
$ pip install pybind11
$ mkdir build
$ cd build
$ cmake ../

Describe the bug
I ran a local test by making a long blockchain with block tools, and adding the blocks 1,000 at a time to the chain. Eventually I got an abort core_dumped, which I'm guessing is one of the C extensions, probably chiapos.

munmap_chunk(): invalid pointer
Fatal Python error: Aborted

Thread 0x00007ff6db29e700 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 306 in wait
  File "/usr/lib/python3.8/queue.py", line 179 in get
  File "/home/mariano/Chia/chia-blockchain/venv/lib/python3.8/site-packages/aiosqlite/core.py", line 95 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007ff6e6f25740 (most recent call first):
[1]    3353766 abort (core dumped)  py.test -s -v 

To Reproduce
Steps to reproduce the behavior:

1. Ask me for the script to run this test
2. Run this test
3. Eventually get a crash

Expected behavior
chiapos should never crash

Desktop (please complete the following information):

• OS: Ubuntu 20.0.4
• CPU: AMD Ryzen 9 3900X

[ 48%] Building C object CMakeFiles/RunTests.dir/src/b3/blake3_dispatch.c.o
gmake[2]: *** [CMakeFiles/RunTests.dir/build.make:76: CMakeFiles/RunTests.dir/tests/test-main.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
[ 51%] Building C object CMakeFiles/ProofOfSpace.dir/src/chacha8.c.o
...
[ 83%] Building ASM object CMakeFiles/chiapos.dir/src/b3/blake3_sse41_x86-64_unix.S.o
[ 86%] Linking CXX executable options_test
[ 86%] Built target options_test
gmake[1]: *** [CMakeFiles/Makefile2:179: CMakeFiles/RunTests.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 88%] Linking CXX shared module ../lib.linux-x86_64-3.8/chiapos.cpython-38-x86_64-linux-gnu.so
[ 90%] Linking CXX executable ProofOfSpace
[ 90%] Built target ProofOfSpace
[ 90%] Built target chiapos
gmake: *** [Makefile:146: all] Error 2
Traceback (most recent call last):
File "setup.py", line 203, in
setup(
File "/usr/lib/python3.8/site-packages/setuptools/init.py", line 144, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib64/python3.8/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib64/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "setup.py", line 38, in run
self.build_extension(ext)
File "setup.py", line 70, in build_extension
subprocess.check_call(
File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j', '6']' returned non-zero exit status 2.

$ rpm -q make cmake automake
make-4.3-6.fc35.x86_64
cmake-3.20.2-2.fc35.x86_64
automake-1.16.3-5.fc35.noarch

Hello,

We need some sanity checking on phase 4 when writing plots, currently when plotting phase 4 it does not check that it has written a plot or indeed anything before finishing and deleting all the temporary files from -t scratch drives.

Scenario, you start plotting and wish to change the destination (it could be full) or did not have your destination drive mounted in Linux. It will enter phase 4 and not write any plots and delete all temp data from scratch drives.

The desired behaviour would be that in phase 4 the plotter checks to see it can write to the 2.tmp file and has enough space before continuing, also there should be logic in place to check it has indeed written a plot before destroying everything from -t scratch drives.

How to reproduce:

I started plotting without having my -d drive mounted, so I just mounted the -d drive at another mount point, copied the 2.tmp file created on my default partition to it, then remounted it as /plots . I did this in p1

start plotting to default filesystem without your plots drive being mounted (in my case /plots)
mount -t auto /dev/SDx (where x is your plot drive) /tmplots

copy the .tmp file from /plots to another drive
cp /plots/plot-k32-2020-11-07-10-05-053affc2a3e4abcd81fd1b61839f0a704862bc239391cc3361229984317cb655.plot.2.tmp /tmplots

umount /tmplots

now mount SDx in the normal /plots place

The plot.2.tmp file is a 0 byte file and writeable for the plotter but it enters phase 4, completes and does not write anything.

    Finished writing C1 and C3 tables
    Writing C2 table
    Finished writing C2 table
    Final table pointers:
    P1: 0x10c
    P2: 0x3747751de
    P3: 0x6b5dca8a1
    P4: 0x9fd8fc784
    P5: 0xd5560d0c7
    P6: 0x10d8d4700f
    P7: 0x14e8db1cfc
    C1: 0x19083f85fc
    C2: 0x190859b908
    C3: 0x190859b9b8

Time for phase 4 = 1424.485 seconds. CPU (66.600%) Sun Nov 8 02:46:20 2020
Approximate working space used (without final file): 286.586 GiB
Final File size: 101.330 GiB
Total time = 60027.865 seconds. CPU (57.370%) Sun Nov 8 02:46:20 2020
Renamed final file from "/plots/plot-k32-2020-11-07-10-05-053affc2a3e4abcd81fd1b61839f0a704862bc239391cc3361229984317cb655.plot.2.tmp" to "/plots/plot-k32-2020-11-07-10-05-053affc2a3e4abcd81fd1b61839f0a704862bc239391cc3361229984317cb655.plot"
02:46:32.722 src.plotting.create_plots : INFO Summary:
02:46:32.753 src.plotting.create_plots : INFO Created a total of 1 new plots
02:46:32.753 src.plotting.create_plots : INFO plot-k32-2020-11-07-10-05-053affc2a3e4abcd81fd1b61839f0a704862bc239391cc3361229984317cb655.plot
zen@plotty:~$ cd /plots
zen@plotty:/plots$ touch me
zen@plotty:/plots$ rm me
zen@plotty:/plots$ ls -altrh
total 913G
drwxrwxrwx 1 root root 0 Mar 6 2017 '$RECYCLE.BIN'
drwxrwxrwx 1 root root 4.0K Oct 19 20:47 'System Volume Information'
-rwxrwxrwx 1 root root 102G Nov 5 18:28 plot-k32-2020-11-05-01-26-0652ac6900ddd31f2dd4db4b4ff4de0cb80ddf94976d68cf24d03b3ed2f6b8ac.plot
-rwxrwxrwx 1 root root 102G Nov 5 18:52 plot-k32-2020-11-05-01-26-14e8601c8956d2e9d871a11a700c1bbfa8c061c67d0b4df1ae00beeca36892ee.plot
-rwxrwxrwx 1 root root 102G Nov 5 19:11 plot-k32-2020-11-05-01-27-58cf324d252c5b3890cd5dd5f6a47e1a9543d5c7b045a5ac1239e9de06c4c8dc.plot
-rwxrwxrwx 1 root root 102G Nov 5 19:14 plot-k32-2020-11-05-01-26-baadb0b06430426de0046647d08110aa709522a2a1c73c32238e9dbec2277966.plot
-rwxrwxrwx 1 root root 102G Nov 6 12:01 plot-k32-2020-11-05-18-28-92e25f2f23ec2d1e306e117cadee433a939094258d8bcc86c68d0d4f023267e9.plot
-rwxrwxrwx 1 root root 102G Nov 6 12:21 plot-k32-2020-11-05-18-52-f46d2635d2be9ee4736b1bd182ed1cfab75dc90d34bbd257c73f1bd7d2d843ab.plot
-rwxrwxrwx 1 root root 102G Nov 7 05:24 plot-k32-2020-11-06-12-01-e0f870c02e1e45a5587c3d2a8ad7fb8f5d84cc04ba9267ff8f4ca9f8260a1678.plot
-rwxrwxrwx 1 root root 102G Nov 7 06:02 plot-k32-2020-11-06-12-21-2665854bc3df001c6ed598cf9c36503118ebc2a6d142c4fc1e43196a89acc4e3.plot
-rwxrwxrwx 1 root root 102G Nov 7 08:12 plot-k32-2020-11-06-13-08-6528ed9031b12db4361fa62eca4d297afee378c71e637ea91ea13b5b175157e7.plot
drwxr-xr-x 27 root root 4.0K Nov 7 10:03 ..
-rwxrwxrwx 1 root root 0 Nov 7 10:05 plot-k32-2020-11-07-10-05-bb223ec0ad01774a4991d01680d3533a9a7ba2ba4db27ae25c9c92b2cc9c3c8d.plot
-rwxrwxrwx 1 root root 0 Nov 7 10:05 plot-k32-2020-11-07-10-05-393d73f049fcb5a7212095143a82306a73099c5dfce633b0f0d6ec11aa65daf7.plot
-rwxrwxrwx 1 root root 0 Nov 7 10:05 plot-k32-2020-11-07-10-05-053affc2a3e4abcd81fd1b61839f0a704862bc239391cc3361229984317cb655.plot
drwxrwxrwx 1 root root 112K Nov 8 13:10 .
zen@plotty:/plots$

p4.txt

I noticed file transfer tops at 1 MiB/s on my gigabit ethernet when a plot's final dir is a sshfs mount. I interrupted the plotting and copied the plot file with cp to the same sshfs mount, which gave me stable 120 MiB/s transfers as usual.

Also while chia was copying the plot, the mount was almost inaccessible, a simple ls would block for minutes. My guess is fs::copy must be writing in very small chunks to the destination.