{"id":213,"date":"2015-05-07T17:39:19","date_gmt":"2015-05-07T16:39:19","guid":{"rendered":"http:\/\/elcep.legtux.org\/?p=213"},"modified":"2015-05-07T22:48:44","modified_gmt":"2015-05-07T21:48:44","slug":"quand-on-a-pas-de-tete-il-y-a-gnu","status":"publish","type":"post","link":"https:\/\/elcep.legtux.org\/?p=213","title":{"rendered":"Quand on \u00e0 pas de t\u00eate il y a GNU"},"content":{"rendered":"<p><a href=\"http:\/\/elcep.legtux.org\/wp-content\/uploads\/2015\/03\/Terminal-icon-shell-linux-unix.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-thumbnail wp-image-229\" src=\"http:\/\/elcep.legtux.org\/wp-content\/uploads\/2015\/03\/Terminal-icon-shell-linux-unix-150x150.png\" alt=\"Terminal-icon-shell-linux-unix\" width=\"150\" height=\"150\" srcset=\"https:\/\/elcep.legtux.org\/wp-content\/uploads\/2015\/03\/Terminal-icon-shell-linux-unix-150x150.png 150w, https:\/\/elcep.legtux.org\/wp-content\/uploads\/2015\/03\/Terminal-icon-shell-linux-unix.png 256w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a>Aujourd&rsquo;hui l&rsquo;un de mes mod\u00e8les que je n&rsquo;arrivais pas \u00e0 faire tourner avec <a title=\"openMole\" href=\"http:\/\/www.openmole.org\" target=\"_blank\">openMOLE <\/a>(tout arrive ) a fini son cycle sur mon vieux phenom II (4 coeur), les autres possibilit\u00e9s sur la grille europ\u00e9enne ou sur CALI \u00e9tant occup\u00e9es sur d&rsquo;autres calculs&#8230; (oui en p\u00e9riode de relecture les calculs s&rsquo;accumulent et il faut \u00eatre organis\u00e9&#8230; je ne le suis pas).<\/p>\n<p>Le fait est que le mod\u00e8le a fini de tourner apr\u00e8s 25 heures et je m&rsquo;aper\u00e7ois que je n&rsquo;ai pas coch\u00e9 la maudite case qui ne garderait que le dernier enregistrement de chaque simulation&#8230; J&rsquo;ai donc \u00e0 faire avec un fichier csv de 5Go &#8230;<\/p>\n<ul>\n<li><a href=\"http:\/\/www.r-project.org\/\" target=\"_blank\">R<\/a>, avec un simple read.csv, ne le mange pas (c.f. package <a href=\"http:\/\/cran.r-project.org\/web\/packages\/data.table\/index.html\" target=\"_blank\">data.table<\/a> peut-\u00eatre?)<\/li>\n<\/ul>\n<p>Je me dis qu&rsquo;on pourrait jouer un peu avec <a href=\"https:\/\/sqlite.org\/\" target=\"_blank\">sqlite<\/a> (histoire de ne pas installer un gros serveur de base de donn\u00e9es). Mais pour \u00e7a il faut supprimer les 6 premi\u00e8res lignes du fichier csv produit par netlogo. On peut le faire gr\u00e2ce \u00e0 un outil bash :<\/p>\n<pre>tail -n +7 partie2\/dion\/resultats\/Dion1952V2_appledGeo_stability-table.csv &gt; partie2\/dion\/resultats\/small2.csv<\/pre>\n<p>Toujours dans la console, on peut v\u00e9rifier que \u00e7a a fonctionn\u00e9 avec head :<\/p>\n<pre>head partie2\/dion\/resultats\/Dion1952V2_appledGeo_stability-table.csv<\/pre>\n<p>qui renvoie<\/p>\n<pre>\"[run number]\",\"export\",\"downerQ\",\"coefUpQuality\",\"chooseMode\",\"demandF\",\"nbPLots\",\"standarDevPop\",\"nbCitys\",\"whatWord\",\"priceMaxWineF\",\"marketColor\",\"[step]\",\"gini-index-reserve\",\"gini-index-patch\",\"meanQualityTotal\",\"meanQualityMountain\",\"meanQualityPlain\",\"DiffExtCentral\",\"nbcentralPlots\",\"meanPatchByNetwork\",\"sum [quality] of patches with [owner != -9999 and alti &lt; 0.1]\",\"sum [quality] of patches with [owner != -9999 and alti &gt;= 0.1]\"\r\n\"3\",\"false\",\"0.5\",\"0.01\",\"false\",\"7000\",\"10\",\"2\",\"10\",\"0\",\"70\",\"false\",\"0\",\"0\",\"0.07083333333333335\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"48\",\"0\"\r\n\"4\",\"false\",\"0.5\",\"0.01\",\"false\",\"7000\",\"10\",\"2\",\"10\",\"0\",\"70\",\"false\",\"0\",\"0\",\"0.04285714285714286\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"42\",\"0\"\r\n\"2\",\"false\",\"0.5\",\"0.01\",\"false\",\"7000\",\"10\",\"2\",\"10\",\"0\",\"70\",\"false\",\"0\",\"0\",\"0.05348837209302328\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"43\",\"0\"<\/pre>\n<p>On passe ensuite \u00e0 sqlite<\/p>\n<pre>\u00a0$ sqlite\r\nSQLite version 3.8.7.4 2014-12-09 01:34:36\r\nEnter \".help\" for usage hints.\r\nConnected to a transient in-memory database.\r\nUse \".open FILENAME\" to reopen on a persistent database.\r\nsqlite&gt; .separator \",\"\r\nsqlite&gt; .header on\r\nsqlite&gt; create table dion(run number,export,downerQ,coefUpQuality,chooseMode,demandF,nbPLots,standarDevPop,nbCitys,whatWord,priceMaxWineF,marketColor,step,giniIndexReserve,giniIndexPatch,meanQualityTotal,meanQualityMountain,meanQualityPlain,DiffExtCentral,nbcentralPlots,meanPatchByNetwork,sum_q_viti_moutain,sum_q_viti_plaine);\r\nsqlite&gt;.import \"partie2\/dion\/resultats\/Dion1952V2_appledGeo_stability-table.csv\" dion\r\nsqlite&gt; DELETE FROM dion WHERE step != 500;\r\nProcessus arr\u00eat\u00e9<\/pre>\n<p>C&rsquo;est un \u00e9chec!!! Et l\u00e0 il y a GNU Linux, et ces outils incroyables qui redonnent espoir dans les moment d&rsquo;abattement !<\/p>\n<pre>awk -F, 'NR == 1 || $13 == \"\\\"500\\\"\"' partie2\/dion\/resultats\/small.csv &gt; partie2\/dion\/resultats\/small500.csv<\/pre>\n<p>ou NR == 1 permet de pr\u00e9server la ligne de header et $13 ==\u00a0\u00bb\\\u00a0\u00bb500&Prime;\\\u00a0\u00bb permet de conserver toutes les lignes qui au niveau de la colonne 13 ont le param\u00e8tre 500, -F , permet de d\u00e9finir le s\u00e9parateur de champs !!<\/p>\n<p>Note : une autre commande super si vous voulez compter le nombre de lignes du fichier !<\/p>\n<pre class=\"lang-sh prettyprint prettyprinted\"><code><span class=\"pln\">cat mycsv.csv <\/span><span class=\"pun\">|<\/span><span class=\"pln\"> wc <\/span><span class=\"pun\">-<\/span><span class=\"pln\">l<\/span><\/code><\/pre>\n<p>Et voil\u00e0 des petites choses \u00e0 ne pas oublier!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Aujourd&rsquo;hui l&rsquo;un de mes mod\u00e8les que je n&rsquo;arrivais pas \u00e0 faire tourner avec openMOLE (tout arrive ) a fini son cycle sur mon vieux phenom II (4 coeur), les autres possibilit\u00e9s sur la grille europ\u00e9enne ou sur CALI \u00e9tant occup\u00e9es sur d&rsquo;autres calculs&#8230; (oui en p\u00e9riode de relecture les calculs s&rsquo;accumulent et il faut \u00eatre &hellip; <a href=\"https:\/\/elcep.legtux.org\/?p=213\" class=\"more-link\">Continuer la lecture de <span class=\"screen-reader-text\">Quand on \u00e0 pas de t\u00eate il y a GNU<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,8],"tags":[32,31,9,33],"class_list":["post-213","post","type-post","status-publish","format-standard","hentry","category-geekeries","category-sma","tag-awk","tag-csv","tag-netlogo","tag-tail"],"_links":{"self":[{"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/posts\/213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=213"}],"version-history":[{"count":12,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/posts\/213\/revisions"}],"predecessor-version":[{"id":238,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=\/wp\/v2\/posts\/213\/revisions\/238"}],"wp:attachment":[{"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elcep.legtux.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}