bugPSPP - Bugs: bug #47090, Running a procudure on a dataset...

 
 

bug #47090: Running a procudure on a dataset with a large number of variables takes an unreasonably long time.

Submitter:  John Darrington <jmd>
Submitted:  Mon 08 Feb 2016 11:38:43 AM UTC
   
 
Category:  Sheet Severity:  9 - Blocker
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  Before 0.10.0
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 01 Mar 2016 08:39:22 AM UTC, comment #11: 

These waiting times are not ideal.  But they are much better than the 5 minutes which we had before, and no worse than what I remember from version 0.8.5

Try the "sheet" branch if you want to see it faster!

I'm therefore closing this item.

John Darrington <jmd>
Group administrator
Tue 01 Mar 2016 08:06:27 AM UTC, comment #10: 

Hi John,

I tried the master git 81f50c24cae1f7b007c3 version which includes your patch. The test below starts in about 16 seconds. I clicked around a bit and could not see immediate problems.

When i try to resize a column in the data window, then this is possible, but it takes about 1-2 seconds after the release of the button, before the change is rendered. This is only the case with 2500 variables. When i delete 2400 variables and have just 100 left, then this is gone. So I guess, the patch works.

Friedrich

Friedrich Beckmann <beckmanf>
Group Member
Tue 01 Mar 2016 05:29:33 AM UTC, comment #9: 

I pushed this patch.

Friedrich, can you please check to see if it causes any ill effects for you?

John Darrington <jmd>
Group administrator
Sun 28 Feb 2016 06:03:33 PM UTC, comment #8: 


> Here is an alternative patch based on Ben's suggestion.

If that also fixes the problem, then it looks great to me.  Thank you!

Ben Pfaff <blp>
Group administrator
Sun 28 Feb 2016 07:49:34 AM UTC, comment #7: 

Here is an alternative patch based on Ben's suggestion.

(file #36498)

John Darrington <jmd>
Group administrator
Thu 25 Feb 2016 09:59:55 PM UTC, comment #6: 

With Johns proposed faster rendering patch I can run the example in approx. 15 seconds. Without the patch it takes one minute for 1500 variables. I stopped the 2500 variables test after 4 minutes.

I attached the patch that John mentioned in an email.

I changed the severity to blocking with respect to the discussion in bug #47266

(file #36466)

Friedrich Beckmann <beckmanf>
Group Member
Sat 20 Feb 2016 05:15:09 PM UTC, comment #5: 

This slow-down should not be a surprise, since commit 8298a4c6e326a39e204ca58d686497758333a102 deletes the optimization that allowed large numbers of columns to be efficient, which I introduced in commit 45861388123559f "pspp-sheet-view: Optimize by making column header widgets lazy."

Ben Pfaff <blp>
Group administrator
Sun 14 Feb 2016 01:28:57 PM UTC, comment #4: 

Hi John,

the mentioned commit is in the pspp-sheet area. As you already tested, it should be gone, once the efficient sheet is alive...

Friedrich

Friedrich Beckmann <beckmanf>
Group Member
Sat 13 Feb 2016 11:03:32 AM UTC, comment #3: 

Using git-bisect  I found that this bug appears to have crept in at
commit 8298a4c6e326a39e204ca58d686497758333a102


John Darrington <jmd>
Group administrator
Fri 12 Feb 2016 07:25:47 PM UTC, comment #2: 

Running the following syntax:

input program.
vector vec(2500).
loop #v = 1 to 2500.
 compute vec(#v) = #v.
end loop.
end case.
end file.
end input program.

descriptives all.


takes:

without a gui:       0.3 seconds
with 0.8.5 gui:       11 seconds
with master HEAD gui:  3 minutes
with a special hacked up gui using the efficient-sheet I'm working on: < 0.5 seconds

I wonder what we've done since 0.8.5 to make it so slow?



John Darrington <jmd>
Group administrator
Wed 10 Feb 2016 05:29:15 PM UTC, comment #1: 

Investigation shows that executing a procedure in the GUI is an operation which is O(n^2) in the number of variables.

I didn't think this was the case :(

John Darrington <jmd>
Group administrator
Mon 08 Feb 2016 11:38:43 AM UTC, original submission:  

In the GUI, running a procedure on a dataset with 2500 variables and 200 cases takes well over 5 minutes.

John Darrington <jmd>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by blp (Posted a comment)
  • -email is unavailable- added by beckmanf (Posted a comment)
  • -email is unavailable- added by jmd (Reporter)
  • -email is unavailable- added by jmd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-01 jmd StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2016-02-28 jmd Attached File- Added 0001-Avoid-multiple-resize-callbacks-when-adding-or-remov.patch, #36498
    2016-02-26 jmd ReleaseNone Before 0.10.0
    2016-02-25 beckmanf Attached File- Added 0001-Johns-faster-rendering-patch.patch, #36466
        Severity5 - Average 9 - Blocker
    2016-02-10 jmd StatusNone Confirmed
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code