Removing duplicate field entries from sorted csv data

Given the following input (cat i.txt), I want to remove duplicate field entries in each of the first three columns and none of the others.

DLORENZ;EDDELAK;BCL;G1;2019-04-01;175 DLORENZ;EDDELAK;BRV/COV;G1;2018-01-31;165 DLORENZ;EDDELAK;BRV/COV;G2;2018-02-28;165 DLORENZ;EDDELAK;BRV/COV;WH;2018-05-29;88 DLORENZ;EDDELAK;BRV/COV;WH;2018-10-02;139 ... 

The input is sorted first on column 1, then on column 2, then on column 3, then on column 4, then on column 5, then on column 6.

That is, from here (cat i.txt | column -s ';' -t)

DLORENZ       EDDELAK            BCL      G1  2019-04-01  175 DLORENZ       EDDELAK            BRV/COV  G1  2018-01-31  165 DLORENZ       EDDELAK            BRV/COV  G2  2018-02-28  165 DLORENZ       EDDELAK            BRV/COV  WH  2018-05-29  88 DLORENZ       EDDELAK            BRV/COV  WH  2018-10-02  139 DLORENZ       EDDELAK            BRV/COV  WH  2019-01-07  140 HELMGBR       GUDENDORF          BCL      G1  2018-04-29  600 HELMGBR       GUDENDORF          BCL      G2  2018-05-28  580 HELMGBR       GUDENDORF          BCL      WH  2018-11-21  600 HELMGBR       GUDENDORF          BOT      G1  2018-07-09  600 HELMGBR       GUDENDORF          BOT      G2  2018-08-06  600 HELMGBR       GUDENDORF          BOT      WH  2019-02-13  600 HELMGBR       GUDENDORF          CHLM     G1  2017-12-14  600 HELMGBR       GUDENDORF          CHLM     G2  2018-01-11  600 HELMGBR       GUDENDORF          CHLM     WH  2018-09-05  550 HKARSTENS     KUDEN              BCL      G1  2019-03-11  255 HKARSTENS     KUDEN              BCL      G2  2019-04-10  255 HSCHLADETSCH  EDDELAK            BCL      G1  2019-03-11  213 HSCHLADETSCH  EDDELAK            BCL      G2  2019-04-08  201 HSCHLADETSCH  EDDELAK            BRV/COV  G1  1979-01-01  218 HSCHLADETSCH  EDDELAK            BRV/COV  G2  1979-01-01  218 HSCHLADETSCH  EDDELAK            BRV/COV  WH  2018-03-13  218 HSCHLADETSCH  EDDELAK            BRV/COV  WH  2018-09-10  160 HWULFF        KUDEN              BCL      G1  2018-02-28  244 HWULFF        KUDEN              BCL      G2  2018-03-28  244 HWULFF        KUDEN              BCL      WH  2018-09-20  190 HWULFF        KUDEN              BCL      WH  2019-03-19  250 HWULFF        KUDEN              CHLM     G1  2018-04-01  244 HWULFF        KUDEN              CHLM     G2  2018-04-29  244 HWULFF        KUDEN              CHLM     WH  2019-03-28  250 JMEIER        EDDELAK            BCL      G1  2018-04-30  360 JMEIER        EDDELAK            BCL      G2  2018-05-28  360 JPETERS       KAISERWILHELMKOOG  CHLM     G1  2018-02-26  65 JPETERS       KAISERWILHELMKOOG  CHLM     G2  2018-03-26  65 JPETERS       KAISERWILHELMKOOG  CHLM     WH  2019-01-18  79 JTHODE        BUCHHOLZ           BCL      G1  2019-03-12  253 JTHODE        BUCHHOLZ           BCL      G2  2019-04-12  253 KMEHLERT      BRUNSBUETTEL       BCL      G1  2018-12-13  79 KMEHLERT      BRUNSBUETTEL       BCL      G2  2019-01-10  119 MMAGENS       BARLT              CHLM     G1  2018-02-13  165 MMAGENS       BARLT              CHLM     G2  2018-03-13  165 MMAGENS       BARLT              CHLM     WH  2018-09-12  136 MMAGENS       BARLT              CHLM     WH  2019-03-14  132 MSCHNEPEL     WINDBERGEN         CHLM     G1  2017-10-09  205 MSCHNEPEL     WINDBERGEN         CHLM     G2  2017-11-02  263 MSCHNEPEL     WINDBERGEN         CHLM     WH  2018-04-10  272 MSCHNEPEL     WINDBERGEN         CHLM     WH  2018-10-25  208 NJUNGE        EDDELAK            BCL      G1  2018-03-07  146 NJUNGE        EDDELAK            BCL      G2  2018-04-04  146 NJUNGE        EDDELAK            BCL      WH  2018-08-06  100 NJUNGE        EDDELAK            BCL      WH  2018-11-14  105 NJUNGE        EDDELAK            BCL      WH  2019-03-12  118 SMOHR         BRUNSBUETTEL       CHLM     G1  2018-04-30  110 SMOHR         BRUNSBUETTEL       CHLM     G2  2018-05-28  110 SMOHR         BRUNSBUETTEL       CHLM     WH  2018-12-18  98 

… I want to arrive at the following output (cat 1fertig.txt | column -s ';' -t):

DLORENZ       EDDELAK            BCL      G1  2019-04-01  175 ----          ----               BRV/COV  G1  2018-01-31  165 ----          ----               ----     G2  2018-02-28  165 ----          ----               ----     WH  2018-05-29  88 ----          ----               ----     WH  2018-10-02  139 ----          ----               ----     WH  2019-01-07  140 HELMGBR       GUDENDORF          BCL      G1  2018-04-29  600 ----          ----               ----     G2  2018-05-28  580 ----          ----               ----     WH  2018-11-21  600 ----          ----               BOT      G1  2018-07-09  600 ----          ----               ----     G2  2018-08-06  600 ----          ----               ----     WH  2019-02-13  600 ----          ----               CHLM     G1  2017-12-14  600 ----          ----               ----     G2  2018-01-11  600 ----          ----               ----     WH  2018-09-05  550 HKARSTENS     KUDEN              BCL      G1  2019-03-11  255 ----          ----               ----     G2  2019-04-10  255 HSCHLADETSCH  EDDELAK            BCL      G1  2019-03-11  213 ----          ----               ----     G2  2019-04-08  201 ----          ----               BRV/COV  G1  1979-01-01  218 ----          ----               ----     G2  1979-01-01  218 ----          ----               ----     WH  2018-03-13  218 ----          ----               ----     WH  2018-09-10  160 HWULFF        KUDEN              BCL      G1  2018-02-28  244 ----          ----               ----     G2  2018-03-28  244 ----          ----               ----     WH  2018-09-20  190 ----          ----               ----     WH  2019-03-19  250 ----          ----               CHLM     G1  2018-04-01  244 ----          ----               ----     G2  2018-04-29  244 ----          ----               ----     WH  2019-03-28  250 JMEIER        EDDELAK            BCL      G1  2018-04-30  360 ----          ----               ----     G2  2018-05-28  360 JPETERS       KAISERWILHELMKOOG  CHLM     G1  2018-02-26  65 ----          ----               ----     G2  2018-03-26  65 ----          ----               ----     WH  2019-01-18  79 JTHODE        BUCHHOLZ           BCL      G1  2019-03-12  253 ----          ----               ----     G2  2019-04-12  253 KMEHLERT      BRUNSBUETTEL       BCL      G1  2018-12-13  79 ----          ----               ----     G2  2019-01-10  119 MMAGENS       BARLT              CHLM     G1  2018-02-13  165 ----          ----               ----     G2  2018-03-13  165 ----          ----               ----     WH  2018-09-12  136 ----          ----               ----     WH  2019-03-14  132 MSCHNEPEL     WINDBERGEN         CHLM     G1  2017-10-09  205 ----          ----               ----     G2  2017-11-02  263 ----          ----               ----     WH  2018-04-10  272 ----          ----               ----     WH  2018-10-25  208 NJUNGE        EDDELAK            BCL      G1  2018-03-07  146 ----          ----               ----     G2  2018-04-04  146 ----          ----               ----     WH  2018-08-06  100 ----          ----               ----     WH  2018-11-14  105 ----          ----               ----     WH  2019-03-12  118 SMOHR         BRUNSBUETTEL       CHLM     G1  2018-04-30  110 ----          ----               ----     G2  2018-05-28  110 ----          ----               ----     WH  2018-12-18  98 

The output will be further processed into a LaTeX input file.

The code I wrote is reasonably straightforward:

First kill the duplicates in column 3 of input, then kill the duplicates in column 2 of the result, then kill the duplicates in column 1 of that result.

It is even efficient enough for my needs (and I can’t come up with anything substantially faster offhand, except for not writing to disk that much). But it is not readable at all.

n="$  (wc -l < i.txt)"  rm -rfv f  mkdir f  cat i.txt > f/i.txt  cd f  while IFS=';' read lbezg rest do     echo "$  lbezg"';'"$  rest" >> 1lw_"$  lbezg" done < i.txt  for file in 1lw_* do     while IFS=';' read lbezg sbezg rest     do         echo "$  lbezg"';'"$  sbezg"';'"$  rest" >> 1lw_2so_"$  lbezg"_"$  sbezg"     done < "$  file" done  for file in 1lw_2so_* do     while IFS=';' read lbezg sbezg impfstoff rest     do         ii="$  (echo "$  impfstoff" | tr -d '/')"         echo "$  lbezg"';'"$  sbezg"';'"$  impfstoff"';'"$  rest" >> 1lw_2so_3impfstoff_"$  lbezg"_"$  sbezg"_"$  ii"     done < "$  file" done  for file in 1lw_2so_3impfstoff_* do     awk -F';' -v OFS=';' ' {if (NR>1) $  3="----"; print $  0}' < "$  file" done > 3fertig.txt  rm 1lw*  while IFS=';' read lbezg rest do     echo "$  lbezg"';'"$  rest" >> 1lw_"$  lbezg" done < 3fertig.txt  for file in 1lw_* do     while IFS=';' read lbezg sbezg rest     do         echo "$  lbezg"';'"$  sbezg"';'"$  rest" >> 1lw_2so_"$  lbezg"_"$  sbezg"     done < "$  file" done  for file in 1lw_2so_* do     awk -F';' -v OFS=';' ' {if (NR>1) $  2="----"; print $  0}' < "$  file" done > 2fertig.txt  rm 1lw*  while IFS=';' read lbezg rest do     echo "$  lbezg"';'"$  rest" >> 1lw_"$  lbezg" done < 2fertig.txt  for file in 1lw_* do     awk -F';' -v OFS=';' ' {if (NR>1) $  1="----"; print $  0}' < "$  file" done > 1fertig.txt  rm 1lw*  ####### the rest is for nice error checking and not strictly necessary  time for i in $  (seq 1 "$  n") do      l1="$  (sed -n "$  i"p < i.txt)"      l2="$  (sed -n "$  i"p < 1fertig.txt)"      echo "$  i"';'"$  l1"'|'"$  i"';'"$  l2"  done | column -s '|' -t > differ.txt 

I wonder how you would go about this?

Blank entries in lastb

My cloud hosted Fedora box mysteriously went down this morning, and I am wondering how paranoid I need to be. The event happened Apr 21, just after 15:00 system time.

In last -x, I can see an apparent reboot (that I didn’t initiate) that did not succeed, resulted in the halt.

swaldman pts/0        64.71.8.198      Mon Apr 22 01:11   still logged in reboot   system boot  4.18.19-100.fc27 Mon Apr 22 01:11   still running swaldman pts/0        64.71.8.198      Mon Apr 22 00:40 - 01:10  (00:30) swaldman tty1                          Mon Apr 22 00:37 - 00:39  (00:02) reboot   system boot  4.18.19-100.fc27 Mon Apr 22 00:36 - 01:11  (00:34) reboot   system boot  4.18.19-100.fc27 Sun Apr 21 15:04 - 01:11  (10:07) swaldman pts/0        198.27.182.181   Sat Apr 13 19:27 - 05:08  (09:40) swaldman pts/0        198.27.182.181   Fri Apr 12 06:50 - 09:04  (02:14) swaldman pts/0        198.27.182.181   Thu Apr 11 00:12 - 20:46  (20:33) 

There are, as there often are, lots of failed ssh attempts which are presumably attacks of some sort. But what is really peculiar is a bunch of blank entries in lastb output at the time of the event:

andrey   ssh:notty    51.77.201.36     Mon Apr 22 00:40 - 00:40  (00:00) andrey   ssh:notty    51.77.201.36     Mon Apr 22 00:40 - 00:40  (00:00) default  ssh:notty    206.189.197.48   Mon Apr 22 00:40 - 00:40  (00:00) default  ssh:notty    206.189.197.48   Mon Apr 22 00:40 - 00:40  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) root     ssh:notty    182.113.224.25   Mon Apr 22 00:39 - 00:39  (00:00) swaldman tty1                          Mon Apr 22 00:37 - 00:37  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00)                                        Thu Jan  1 00:00 - 00:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:02 - 15:02  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:01 - 15:01  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) hydra    ssh:notty    119.197.77.52    Sun Apr 21 15:00 - 15:00  (00:00) root     ssh:notty    218.92.0.167     Sun Apr 21 15:00 - 15:00  (00:00) 

Has anyone encountered these kinds of blank entries?

Other than this, I can find no evidence of an intrusion. The optimistic view is that something managed to cause a crash but did not succeed beyond that. Obviously a pessimistic view is that an intruder covered her tracks (except for these blank entries in lastb). I’m sure the wise thing to do is to err on the side of pessimism, but if anyone has encountered blank entries like this before, I’d appreciate any insight or experience.

Many thanks!

Finding duplicated entries

Since the Esteemed Academia wants me to learn Haskell, and Haskell seems just weird for me, I thought that to train some basic Haskell skills I’d write a simple script I now need in Haskell.

This is what the script is supposed to do… I have a file that looks like this:

oo R j j ai            I'm a comment kd oo stack j a 

I needed to extract duplicated elements from this file with lines they appear on. The catch is that the tab character denotes comments; so I'm a comment should not be considered here and in particular, this a from I'm a comment should not match this a that appears as the last element from these sample lines.

If fed this particular sample file, my script should report finding duplicate elements: oo on lines 1 and 4 and j on lines 2, 3 and 5. I believe I wrote a script that does this correctly.

Here it is:

import qualified Data.Map as Map import Data.List import Data.Function  main = do   input <- fmap preprocess getContents   printDupes $   findDupes input  preprocess input = let     untabbedLines = map (takeWhile (/= '\t')) $   lines input     wordsedLines = map words untabbedLines     numberedLines = zip wordsedLines [1..]     numberedWords = concat $   map indexDown numberedLines where       indexDown (words', index) = map (flip (,) $   index) words'   in numberedWords  findDupes entries = let     occurences = Map.fromListWith (flip (++)) $   map (fmap (:[])) entries   in Map.filter ((>1).length) occurences  printDupes :: Map.Map String [Int] -> IO() printDupes dupes = let     showPositions positions = intercalate ", " $   map show positions     printDupe (dupe, positions) = putStrLn $   "Duplicated element " ++ dupe ++ " found on positions: " ++ showPositions positions   in let     sortedDupes = sortBy (compare `on` snd) $   Map.assocs dupes   in mapM_ printDupe sortedDupes 

The task is simple, but I guess it’s OK for learning how to code in Haskell…

So could you kindly review this self-imposed exercise? What could’ve been done better than I did? Simpler? Shorter? I’m sure much, but what precisely? Also how to improve readability?

fclose is causing a crash and works for less than 10k entries when commented. Could someone comment what could be wrong with the below program

Here goes the program    #include <stdio.h>   #include <string.h>   #include <stdlib.h>     void main() {         FILE *fp;         char str[200];         char *tempstr;         char close[65000][10];         char open[65000][10];         char station[65000][20];         static int count;         tempstr=(char *)malloc(200);         bzero((void *)str,200);         system("$  SHELL parselog.sh 52799272_LOG1");         do {               system("head -n 7 tempfile > temp");               fp=fopen("temp","r");               while((fgets(str,120,fp)) != NULL) {                         if(strstr(str,"OPEN")!= NULL) {                                 strcpy(open[count],strtok(str," "));                                 printf("%s\n",open[count]);                         } else if(strstr(str,"CLOSE")!= NULL) {                                 strcpy(close[count],strtok(str," "));                                 printf("%s\n",close[count]);                         }                         else if (strstr(str,"YOURHOSTNAME:")!= NULL) {                                 printf("%s\n",str);                                 bzero((void *)tempstr,200);                                 strcpy(tempstr,str);                                 tempstr = strtok(tempstr," ");                                 printf("tempstr:%s\n",tempstr);                                 tempstr = strtok(NULL," ");                                 printf("tempstr:%s\n",tempstr);                                 strcpy(station[count],tempstr);                                 printf("%s\n",station[count]);                                 count++;                                 bzero((void *)str,200);                                 break;                         }                         //      printf("%s\n",str);                         bzero((void *)str,200);                 }                 system("cp tempfile tmp");                 remove("tempfile");                 system("tail -n +9 tmp > tempfile");                 fclose(fp); //causing the crash when uncommented                 fp=fopen("tmp","r");                 fgets(str,120,fp);                 fclose(fp);                 remove("temp");                 remove("tmp");           }while(str != NULL); 

}

The fclose in line#49 is causing the issue when uncommented it crashes right after parsing 2 entries and when commented it crashes stating there are too many open files.

case 1: free(): invalid pointer Aborted (core dumped)

case 2: cp: cannot create regular file ‘tmp’: Too many open files sh: 1: 1: Too many open files Segmentation fault (core dumped)

Could someone point what is wrong with my program?

MongoDB db._____.find() is only returning entries when I am using a different database in the shell

I am designing a dynamic server using node.js and express. For persistence I am using MongoDB.

I have a register page that takes the form data and enters it into the database known as user.

A login page that checks the users password/username with that on the database, if it matches let them in.

I am also making an email esque service that allows communication between different accounts, I am doing this using a second database called message.

I have added a few entries to the users database and have logged in successfully, however after attempting to add the secondary mongodb database it is no longer working in a way that I would expect.

Firstly when adding data to the message database, attempting to view the contents leads to an empty response. Secondly, when using the command use users in the mongo shell and then db.users.find() it returns nothing. However if I go into use message and then repeat the command db.users.find() it returns all the data I would expect.

Using the command db.message.find() returns nothing regardless of what use I use.

This is how I am connecting to the databases:

const mongoose = require('mongoose').Mongoose; const User = require('./models/schema.js'); const Message = require('./models/msgSchema.js')  // connecting to the MongoDB data base var users = new mongoose(); users.connect('mongodb://localhost/users')  // connecting to the MongoDB data base var users = new mongoose(); users.connect('mongodb://localhost/users') 

My 2 schemas are schema.js && msgSchema.js

msgSchema:

const mongoose = require('mongoose'), Schema = mongoose.Schema;  // Connect to database mongoose.connect('mongodb://localhost/message');  var msgSchema = new mongoose.Schema ({     user: {         type: String,         required: true,     },     message: {         type: String,         required: true,     } });  var Message = mongoose.model('Message', msgSchema); module.exports = Message; 

schema:

const mongoose = require('mongoose'), Schema = mongoose.Schema; const bcrypt = require('bcrypt');  // Connect to database mongoose.connect('mongodb://localhost/users');  var userSchema = new mongoose.Schema ({     username: {         type: String,         unique: true,         required: true,         trim: true     },     password: {         type: String,         required: true,     },     email: {         type: String,         required: true     } });  var User = mongoose.model('User', userSchema); module.exports = User; 

There is more in schema but it is just hashing/authentication so not important

My POST for register

app.post('/register', (req, res) => {     if(req.body.email && req.body.username && req.body.password) {         var userData = {             email: req.body.email,             username: req.body.username,             password: req.body.password,         }          User.create(userData, function(err, user) {             if(err) {                 return console.log(err.message);             } else {                 console.log("Adding data to users was successful...");                 req.session.userId = user._id;                 return res.redirect('/login');             }         });     } }); 

My POST for the messaging system

app.post('/MembersArea', (req, res) => {     console.log("POSTS FINE");      if(req.body.message && req.body.user) {         var userMessage = {             message: req.body.message,             user: req.body.user,         }          Message.create(userMessage, function(err) {             if(err) {                 console.log(err.message)                  return res.redirect('/MembersArea');             } else {                 console.log("Adding data to message was successful...");                 return res.redirect('/MembersArea');             }          })     } }) 

Duplicate entries sometimes being appended to a multi-line text field

I have a list in SharePoint 2013 that contains a multi-line text column for comments that is set to append new entries. Sometimes, when user X tries to append a comment, instead of the new comment the field posts 1-8 copies of the previous comment but under user X's name.

When I look at the version history for the list item, there are blank versions for each duplicate item created. Oddly, I have only seen this occur for this one user, and she has confirmed that she is using the list in the same way as everybody else on her team.

The list has a small customization where it pulls the item Id from the query string and displays it with some HTML in a content editor web-part above the item detail page, but other than that it is straight OOTB.

Has anybody seen behavior like this from SharePoint 2013 multi-line text boxes, and if so, were you able to correct it?

Deleted an EFI entry by mistake trying to cleanup the entries

I’ve been distrohopping for some time on my Thinkpad, and it created a lot of entries in the bootmenu. My thinkpad shows this entries when I choose to select a device to boot. I wanted to delete this entries using efibootmgr but I mispelled the bootnum and deleted something that I was not soposed to delete. I deleted the entry “Boot0001 Boot Menu FvFile(126a762d-5758-4fca-8531-201a7f57f850)” I tried to create the entry again, but it puts disk and partition information that was not there in the original entry. How can I create the entry with only the “FvFile(..)” ?

Thanks

Removing numerous ‘homepage’ entries from Contacts.app with AppleScript extremely slow

A syncing issue with Outlook left me with hundreds, if not thousands of duplicate contacts. After managing to merge duplicates without Contacts crashing, I was left with 177 contacts, most of which with many repeat homepage entries. Rather than dying of boredom removing these by hand, I put together some AppleScript to do this for me, thinking that this would take a few minutes. It’s been a week now – the script starts well enough but soon slows down continually and also takes more and more memory from the system, until the spinning beachball of doom appears halting the script. One issue is that I seem to only be able to delete a contact’s urls one at a time in sequence, instead of all at once.

So the question is, what have I got wrong making this script near useless? Could it have something to do with iCloud syncing? Or is AppleScript inherently inefficient? (The constant saving is there because of the random times the script would cease functioning.):

tell application "Contacts"     activate     with timeout of 72000 seconds         set myPeople to people         set numPeople to (count of myPeople)         repeat with i from 1 to numPeople             set myGuy to item i of myPeople             set myGuyName to get name of myGuy             set personUrls to (the urls of myGuy whose value contains "outlook")             set urlNum to count of personUrls             if urlNum > 0 then                 repeat with j from urlNum to 1 by -1                     log ((time string of (current date)) & " – [" & i & "/" & numPeople & "] " & myGuyName & " (" & j & "/" & urlNum & "): " & (the label of item j of personUrls))                     delete item j of personUrls                     save                 end repeat             else                 log "No problematic URLs found for " & myGuyName             end if             if note of myGuy is not missing value then set note of myGuy to ""         end repeat         save         log "Final save"     end timeout     return end tell