bugGNU Parallel - Bugs: bug #44961, perl hex recipient

 
 

bug #44961: perl hex recipient

Submitter:  Ole Tange <tange>
Submitted:  Wed 29 Apr 2015 07:55:39 PM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 18 Mar 2022 09:27:39 PM UTC, comment #5: 

How about just:

  (echo 'while(<STDIN>) { print; }
  _END_';
   <num30000 perl -e 'while(@ARGV){sysseek(STDIN,shift,0)||die;$left=shift;while($read=sysread(STDIN,$buf,$left>32767?32767:$left)){$left-=$read;syswrite(STDOUT,$buf);}}' 0 0 0 168894) |
     ssh server perl

Or

  ... | (echo 'while(<STDIN>) { print; }
  _END_'; cat ) | ssh server perl

No need for hex decoder: Just start by printing the perl code, followed by the data on standard input - either as 'cat' (to pass stdin) or as part of '--pipepart'.

Ole Tange <tange>
Group administrator
Thu 17 Mar 2022 07:53:19 PM UTC, comment #4: 

Dont use base64, but hex.

The hex-decoder can be made so it does not need to be quoted.

Ole Tange <tange>
Group administrator
Tue 08 Dec 2015 12:15:46 AM UTC, comment #3: 

It does not make sense to compress:

null() { head -c $1 /tmp/seq |  ssh grb.pi.dk 'wc -c'; }
bz() { head -c $1 /tmp/seq | bzip2 | ssh grb.pi.dk 'bzcat|wc -c'; }
export -f null
export -f bz
parallel  -j2 --joblog - {3} {1}0 ::: 100000 300000 1000000 3000000 ::: {1..10} ::: bz null

Only at 30000000 bytes is compression when running on a 10 Mbps line, and it is safe to assume the line is mostly faster.

Ole Tange <tange>
Group administrator
Wed 29 Apr 2015 08:31:32 PM UTC, comment #2: 

# Input:                                                                                                                     
#   zipper space Base64 16384*space \n                                                                                       

my($zip,$base64);
{
    my @b;
    my $kb8=" "x8192;
    # Read most in blocks of 8 KB                                                                                            
    while(sysread(STDIN,$buf,8192)) {
        if($buf eq $kb8) { last; }
        push @b, $buf;
    }
    ($zip,$base64) = split / /,join "",@b;
    # Read the final partial block byte by byte                                                                              
    # to avoid reading too much from STDIN                                                                                   
    while(sysread(STDIN,$buf,1)) {
        if($buf eq "\n") { last; }
    }
}
print $zip;
$dec = decode_base64($base64);
if($zip) {
    # filter $dec through zipper                                                                                             
    $dec = $dec;
}
eval $dec;

Ole Tange <tange>
Group administrator
Wed 29 Apr 2015 08:05:48 PM UTC, comment #1: 

Problem: <> reads 8kb block.

Ole Tange <tange>
Group administrator
Wed 29 Apr 2015 07:55:39 PM UTC, original submission:  

Make a perl wrapper that:

reads 1 line of Base64 encoded perl script. Executes the script.

This will make it possible to ignore the csh limitation of word length and the bash limitation of line length.

It should be compatible with --pipe, so the wrapper should pass the rest of STDIN to the --pipe command.

Does it make sense to bzip2 the script when running remote? How long does it take to bzip2/bunzip2 compared to transfer time? Without the bzip2 it will make it easier to simply run the same unpacker locally and remotely. Alternatively the Base64 line can be prepended with 'bzip2' if it needs to be bunzipped.




Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-03-17 tange Summaryperl base64 recipient perl hex recipient
    2015-04-29 tange Severity3 - Normal 1 - Wish

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code