Refresh writing progress bar to new line of console about konsole HOT 18 OPEN

@lilasquared just got on the train going home, will take a look at that now

from konsole.

hi @lilasquared Do you want to block (await) the processing of each partition before starting the next partition? If the thing that calls the code you supplied is doing while (workNotFinished) { // call the foreach then that might explain the screenshot you supplied, i.e. at low workload your perform work does the work in 1 call, while under high workload if your performWork call does a lot of work and has to batch the work between ReportProgress calls, then that might explain whats happening.

I've got concurrency tests for various threading scenarios, which I believe should be sufficient to cover your use case, so would first look for a logic error before assuming the progress bar is at fault.

[All the concurrency tests are here] (https://github.com/goblinfactory/konsole/tree/master/Konsole.Tests.Slow)

I would suggest you look at my unit test (when I find it) and tell me if that's good enough? If it needs to be hardened then I'd happy to do that.

My own demo code that runs seperate threads updating progress bars is done as follows

        public static void ProgressBarDemo(IConsole con)
        {
            con.Clear(ConsoleColor.Black);
            con.WriteLine("Press enter to start");
            con.ForegroundColor = ConsoleColor.White;
            var r = new Random();
            var dirs = TestData.MakeObjectNames(r.Next(20) + 3);

            var tasks = new List<Task>();
            var bars = new List<ProgressBar>();
            foreach (var d in dirs)
            {
                var dir = new DirectoryInfo(d);
                var files = TestData.MakeFileNames(r.Next(100) + 10);
                if (files.Length == 0) continue;
                var bar = new ProgressBar(con, files.Count());
                bars.Add(bar);
                bar.Refresh(0, d);
                tasks.Add(new Task(() => ProcessRealFiles(d, files, bar)));
            }
            Console.ReadLine();
            foreach (var t in tasks) t.Start();
            Task.WaitAll(tasks.ToArray());
            con.ForegroundColor = ConsoleColor.Yellow;
            con.WriteLine("finished.");
        }

I have re-looked at the concurrency tests and I see that what the tests do is use a seperate thread per window that the progress reporting is done on. That is not a requirement, and I have working demo's (e.g. the animated gif on the home page) which is deliberately a separate thread per progress bar, so the tests are lacking a good example covering that use case.

I believe the code should work with seperate threads per window, but unfortunately don't have an acceptance test proving that. If you can supply a minimalist example reproducing the error I'd be very grateful and would probably use your sample as a basis, or inspiration for another acceptance test.

please let me know how you get on,
hopefully this help?

best regards,
Alan

from konsole.

@lilasquared One quick thought, could your problem be caused by the window scrolling WHILE you're updating the screen from multiple threads?
i.e. similar to this issue which was raised, and which I am currently not going to fix or support for the reasons mentioned in this issue ticket

Cursor keeps in bottom of console, causing the scroll to be Stuck #11

If you are running so much work that you're scrolling off the screen that would explain the behavior.

The workarounds available are:

1. Split the work into batches and report the overall progress of the batches, not the individual items.
2. use the windowing API that's provided to open two or two or three full height console windows next to each other and spread the progress bars across each window. See my comments in the mentioned issue ticket. (Three windows side by side with height of 30 rows, would accommodate up to 90 separate threads / units of work, without scrolling.)

If the scrolling is an issue, you'll need to determine the available screen height and make sure you have enough space to create the scrollbars, and split the console into 2, 3 or 4 windows appropriately.

If you have too many progressbars to fit in one window, such that the window starts scrolling, then using progress bars will no longer provide a happy UX experience imho. If that's the case then I would recommend you possibly do something like a 'roll up', ... grouping large collections of units of work into batches, and doing a group by total for each batch, reporting via progress bars the total overall progress, e.g. batchProgress = batch.Sum(b=> b.Progress) / b.Sum(b=> b.Max) ... very roughly.

You could also separately show a progress bar indicating the total number of finished files.

What do you think?

best of luck

Alan

p.s. just realised batch is the wrong word to use above as it suggests you might be processing them one batch at a time, grouping/group is more appropriate if all the groups of unit of works, are being run simultaneously.

from konsole.

@goblinfactory thank you for the detailed replies. To answer your first question, each partition is processing sequentially. I am creating a progress bar, then firing off the work that may take 1 second or may take 5 minutes, then going to the next partition after it is completed. The work itself (inside of PerformWork is a Parallel.ForEach so the progress bar could be getting updated from multiple threads from within the PerformWork method. There is no while loop or other control flow outside of the foreach. The reason I am not using a single progress bar is because each iteration of the loop contains some logging that also goes to the console. So each round of iteration looks something like this

               (68 %) ###############################################
[09-26-2018 20:25:54] Info  - Begin Partition X
[09-26-2018 20:25:54] Debug - <some sql query>
[09-26-2018 20:25:54] Info  - Begin Work
... (during this time the progress bar is updating)
[09-26-2018 20:25:54] Info  - End Work
[09-26-2018 20:25:54] Info  - Begin Writing Results
[09-26-2018 20:25:56] Info  - End Writing Results
[09-26-2018 20:25:54] Info  - EndPartition X

Due to the logging the progress bar needs to be recreated for each partition otherwise the scrolling issue you mentioned would be happening (which did happen the first time i split the work into partitions).

As for the UX part of this, the progress bars are just there for me to know how long this process is roughly taking. since each partition is not an equal amount of work it was just useful info for me. I think down the road this will just be logged to a database somewhere with a GUI that shows the progress.

I will definitely work on the sample project that recreates the scenario, but the basics of it is - only 1 progress bar is ever active at a time, and I am logging to the console between when the bar is created and when it is updated. Another note would be that the first 20 or so iterations work as expected. Again, I will try to get a better sample project that shows it directly.

Thanks again for taking the time to look into this!

from konsole.

here is a link to a sample project that replicates the behavior. You will notice it starts going wild around iteration 50 or so.

https://github.com/lilasquared/konsole-bug

from konsole.

Hi @lilasquared I've cloned the project and run it, and it runs fine on Windows.
What OS are you running on? Konsole is not a netstandard project.
Originally i could not port to full netstandard since critical key portions of Console have or had not been implemented.

checkout : https://source.dot.net/#System.Console.Tests/WindowAndCursorProps.cs

For example:

    [Fact]
    [PlatformSpecific(TestPlatforms.AnyUnix)]  // Expected behavior specific to Unix
    public static void BufferWidth_SetUnix_ThrowsPlatformNotSupportedException()
    {
        Assert.Throws<PlatformNotSupportedException>(() => Console.BufferWidth = 1);
    } ```

and there are others.

from konsole.

That project should be net framework, I'm running on windows 7

from konsole.

                                         (100%) ################################################################################################################
Begin Partition 53
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
End Partition 53
                                         (100%) ################################################################################################################
Begin Partition 54
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
End Partition 54

                                         (100%) ################################################################################################################
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
Some other logging
End Partition 55

Begin Partition 56
Some other logging
Some other logging
Some other logging
                                         (1  %) ##
                                         (10 %) ############
                                         (12 %) ##############
                                         (3  %) ####
                                         (7  %) #######
                                         (5  %) ######
                                         (17 %) ###################
                                         (9  %) ##########
                                         (23 %) ##########################
                                         (25 %) #############################
                                         (28 %) ###############################
                                         (30 %) #################################
                                         (31 %) ##################################
                                         (33 %) ####################################
                                         (20 %) ######################
                                         (18 %) #####################
                                         (38 %) ###########################################
                                         (15 %) ################
                                         (43 %) ################################################
                                         (21 %) ########################
                                         (46 %) ###################################################
                                         (37 %) #########################################
                                         (50 %) ########################################################
                                         (51 %) ##########################################################
                                         (54 %) ############################################################
                                         (40 %) #############################################
                                         (34 %) #######################################
                                         (58 %) ##################################################################
                                         (61 %) ####################################################################
                                         (62 %) ######################################################################
                                         (47 %) #####################################################
                                         (43 %) #################################################
                                         (68 %) #############################################################################
                                         (69 %) ##############################################################################
                                         (56 %) ##############################################################
                                         (56 %) ###############################################################
                                         (74 %) ##################################################################################
                                         (77 %) #######################################################################################
                                         (80 %) #########################################################################################
                                         (81 %) ##########################################################################################
                                         (82 %) ############################################################################################
                                         (85 %) ###############################################################################################
                                         (72 %) ################################################################################
                                         (75 %) #####################################################################################
                                         (64 %) ########################################################################
                                         (67 %) ###########################################################################
                                         (87 %) #################################################################################################
                                         (88 %) ###################################################################################################
                                         (91 %) #####################################################################################################
                                         (95 %) ###########################################################################################################
                                         (93 %) ########################################################################################################
                                         (93 %) #########################################################################################################
                                         (98 %) #############################################################################################################
                                         (100%) ################################################################################################################
                                         (100%) ################################################################################################################
Some other logging
Some other logging
Some other logging
End Partition 56

i stuck in a breakpoint and here is it going bonkers for partition 56

from konsole.

I may have to setup an AppVeyor build server and you create a pull request for a branch containing a failing unit test so that we can rule out machine differences, and get AppVeyor to build and run tests, otherwise I might not be able to reproduce, and-or won't know when I've fixed your problem.

Also need to prove that the problem exists outside of your machine.

from konsole.

Just gave it a shot on another machine - windows 7 VM on my macbook pro, same problem, though it was around the 40th iteration instead. I cloned down the code and i'm trying to step through and see if anything jumps out at me

from konsole.

@goblinfactory i think it is Console.BufferHeight . On the iterations that are messing up Console.CursorTop is stuck at 499, and the default Console.BufferHeight = 500. If i set Console.BufferHeight = 2000 in aWriter constructor, it works. Obviously not the permanent solution but it resolves the behavior

from konsole.

I just checked and the buffer for my console is actually set to 500 so I can modify those defaults, maybe this shouldn't be Konsole's problem at all

from konsole.

The difficulty is always creating a unit test so we can fix it, and have proper regression test on top of the fix. I created the MockConsole for unit testing most of the behaviours, I can probably extend the behaviour of the mockConsole to include buffer and scroll height behaviour.

If a progress bar scrolls off the screen then unfortunately annoyingly we still have to update it on refresh in case someone manually scrolls back up, eurgh ,so a simple shortcut of not updating when scrolled off screen is possibly not correct?
In the meantime if you can increase your buffer that's a temporary workaround?

I'll leave this issue and the discussion open, as it's quite valuable and we might decide to do something about this.

A side note: I want to find a clever way to migrate this to dotnet standard at some point even though some console features are not supported.

from konsole.

The screen size and console buffer are two different things. I think if the console output exceeds the buffer size then whatever was at the beginning is lost entirely, so any progress bar that existed outside of the buffer can't be updated anyway. If you tried to do the update you would be overwriting some other lines on the console because the bar no longer exists at that Y position.

If you're interested you should checkout https://github.com/PawelGerr/Thinktecture.Abstractions, they have a console abstraction you might be able to swap in so that you aren't maintaining your own.

If you use a very small buffer, like 10 and do something like this you will see what I mean

Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
var progressBar = new ProgressBar(100);
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");
Console.WriteLine("Some other logging");

In this scenario you will see the empty space created for the progress bar, but when you try to refresh it, the new lines will be refreshed below that empy space overwriting one of the "Some other logging" statements.

from konsole.

@lilasquared Pull requests are always welcome! :)
Using IConsole from thinktecture is a good idea, I will consider that so that Konsole, especially the Windows in Konsole can be used as pluggable ThinkTecture.IConsole replacements, given that thinktecture abstractions will probably be more widely used, or we should encourage wider usage.

Please also take a look at Goblinfactory.ProgressBar. I have moved the progress bar to a standalone project, also ported it to .net standard 2. I'm sure it will have the same issues you've described above, so a pull request there, specifically to address this bug will be most appreciated.

from konsole.

@lilasquared I'm going to close this issue, since progress bar has been moved to a seperate github project, with a seperate nuget package.
That project is now here: https://github.com/goblinfactory/progress-bar/

from konsole.

reopened: in case someone wants to work on this project, so that they can see what are still outstanding issues, regardless of what has been ported to other projects.

from konsole.

ProgressBar is going to be worked on quite a bit in upcoming version 7 and 8 of Konsole. Dropping a comment here to keep this conversation fresh. There are some interesting trade-offs with different ways to address this and other "buffer" overrun issues. What do to when buffer end is reached etc. chat later...cheers, Alan

from konsole.