User Tools

Site Tools


wiki:installation_of_the_new_toh_functionality

Installation of new toh in the OpenWrt wiki

Find below the to-do's for OpenWrt-wiki admins and the OpenWrt community.

OpenWrt Wiki-Admins

Install ''Structured Data'' plugin

Done

Purpose: Provides database capabilities. dataentries contain the data, datatables use this data. For further info how this works, see link below.

  1. Dokuwiki: install "Structured Data" plugin → https://www.dokuwiki.org/plugin:data
  2. Dokuwiki: install "sqlite" plugin → https://www.dokuwiki.org/plugin:sqlite
  3. sudo apt-get install php5-sqlite
  4. done, Structured data plugin should work now.

Install ''Datatemplate'' plugin

Done

Purpose: Add-on to the Structured data plugin. Datatemplate provides usage of templates for presenting data in different ways than the data plugin itself would allow.

  1. Dokuwiki: install "datatemplate" plugin → https://www.dokuwiki.org/plugin:datatemplate
  2. done, datatemplate plugin should work now.

Install ''hidepages'' plugin

Purpose: Hide dataentry pages from being displayed in search results and siteindex.

  1. Dokuwiki: install "hidepages" plugin → https://www.dokuwiki.org/plugin:hidepages
  2. done, hidepages plugin should work now.

Install ''Bureaucracy'' plugin

Done

Purpose: Provides forms for creation of new dataentry pages / new device pages

  1. Dokuwiki: install "Bureaucracy" plugin → https://www.dokuwiki.org/plugin:bureaucracy
  2. done, Bureaucracy plugin should work now.

Install ''Folded'' plugin

Done

Purpose: Provides capability to fold away big chunks of data for better, more compact structure of pages. The complete big data chunk is shown, when the user clicks on a link (can be named anything you like).

  1. Dokuwiki: install "Folded" plugin → https://www.dokuwiki.org/plugin:folded
  2. done, Folded plugin should work now

Add datapages to OpenWrt wiki

Done

  1. uncompress to <wikiroot>/toh/hwdata
    • Make sure that the result is like <wikiroot>/toh/hwdata/<brand>/*.txt
    • Make sure that owner, group and access rights are set correctly for all imported directories and files, otherwise datapages will not be editable
    • hwdata can be named anything you like and can be placed anywhere you like. <wikiroot>/toh/hwdata is a first suggestion. You could also do like this: <wikiroot>/techdata instead, it doesn't matter. Therefore: Do as you like or how it makes most sense to keep the wiki structured and modular.
  2. Get dataentry pages in the database
    1. Update dokuwiki indexadmin / Searchindex manager (install this plugin if not already present) / Update Index (takes about 25mins on a rpi2 for 950 datapages alone; new indexing of the complete OpenWrt wiki should take quite a bit more… → Do Index update in low traffic times, e.g. around 0300-0500 CET; Note: Requires js-capable browser-window to be open during the whole index-update)
    2. Do a complete download of the newly added pages/namespace, e.g. via a script and wget, see https://www.dokuwiki.org/plugin:data#how_to_refresh_data; Should possibly also be done in low traffic times, e.g. around 0300-0500 CET)
  3. Database cleanupadmin / Data Plugin: Clean up database / Clean database (should return with 0 entries removed, since the db is just newly set up). Even if it does nothing, it's good practice to do it each time the index gets rebuild or datapages are edited / moved / renamed / deleted from outside of the wiki (i.e. not via the dokuwiki edit page and hence being out of reach of the dataplugin).
  4. Define data-plugin Field Type Aliases (needed for dropdowns in dataentry editor) → admin / Data Plugin: Field Aliases
    Type Alias        Valid values
    availability      ¿, Available, Available 2015, Available 2016, Discontinued, Discontinued 2010, Discontinued 2011, Discontinued 2012, Discontinued 2013, Discontinued 2014, Discontinued 2015, Discontinued 2016
    bootloader        ¿, U-Boot, CFE, Adam2, ARMboot, Bootbase, Coreboot, brnboot, JBoot, MyLoader, NetBoot, PP_Boot, PSPBoot, RedBoot, RGLoader, ThreadX, VxWorks, Yamon, other
    brand             3Com, Abicom International, Actiontec, ADB, ADI Engineering, Airlink101, ALFA Network, Allnet, Arduino.cc, Arcadyan / Astoria, AsiaRF, Asus, AVM, BT, Belkin, Buffalo, Cisco, COMFAST, Compex, Comtrend, Cubitech, D-Link, EasyAcc, EasyLink, Edimax, ELECOM, EnGenius, Evaluation boards / unbranded boards, Fon, Gateway, Gateworks, Gigaset, GL.iNet, HiWiFi/Gee, HooToo, Huawei, Inventel, JCG, Kingston, Lemaker, Lenovo, Linksprite, Linksys, MeLE, Meraki, Mercury, MikroTik, NetComm, Netgear, Nexx, Olimex, OMYlink, Open-Mesh, PC Engines, Pirelli, Planex, Pogoplug, Poray, Qemu, Raspberry Pi Foundation, Rosewill, SMC, Sagem, SamKnows, Samsung, Scientific Atlanta, Seagate, Siemens, Sitecom, SmartRG, SolidRun, STORYLiNK, T-Com / Telekom, TP-Link, Telsey, Tenda, Texas Instruments, Thomson, Trendnet, Turris CZ.NIC, US Robotics, Ubiquiti, Upvel, VONETS, Western Digital, WIZnet, Xiaomi, Xunlong, ZBT, ZTE, ZyXEL
    devicetype        WiFi Router, Router, Travel Router, Travel Router - Battery powered, Range Extender, Single Board Computer, Modem, NAS, unknown, other
    instructionset    ¿, ARM, AVR32, MIPS32, MIPS64, MIPS IV, PPC, x86, x86_64, other
    mbflash           ¿, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 64NAND, 128NAND, 256NAND, 512NAND, 1024NAND, 2048NAND, 4096NAND, 8192NAND, more than 8GB, more than 8GB NAND, SD, microSD, microSDHC
    mbram             ¿, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, more than 4GB
    modem             ¿, Yes, No, ADSL, ADSL2+, ADSL2+ Annex A, ADSL2+ Annex B, VDSL2, xDSL, DOCSIS, LTE, Mobile Wireless/Cellular network, Powerline
    numbutton         ¿, -, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, more than 20
    numdetachantenna  ¿, -, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, more than 10
    numethport        ¿, -, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, more than 20
    numled            ¿, -, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, more than 20
    numsataport       ¿, -, 1, 2, 3, 4, 5, 6, more than 6
    releasecurrent    ¿, -, 0.9, 7.06, 7.09, 8.09, 8.09.2, 10.03, 10.03.1, 12.09, 14.07, 15.05, 15.05.1, DD trunk, external image
    releasesince      ¿, -, 0.9, 7.06, 7.09, 8.09, 8.09.2, 10.03, 10.03.1, 12.09, 14.07, 15.05, 15.05.1, external image
    subinstructionset ¿, ARMv4, ARMv5, ARMv6, ARMv7, ARMv7-A, MIPS32 4K/E series, MIPS32 24K/E series, MIPS32 34K series, MIPS32 74K series, MIPS32 1004K series, MIPS32 1074K series, other
    target            ¿, -, adm5120, adm8668, ar7, ar71xx, at91, atheros, au1000, avr32, bcm53xx, brcm-2.4, brcm2708, brcm47xx, brcm63xx, cns21xx, cns3xxx, cobalt, ep93xx, imx23, imx6, iop32x, ipq806x, ixp4xx, gemini, kirkwood, lantiq, magicbox, malta, mcs814x, mpc52xx, mpc83xx, mpc85xx, mvebu, mxs, netlogic, octeon, olpc, omap, orion, oxnas, ppc40x, ppc44x, ps3, pxa, ramips, rb532, realview, rdc, sibyte, sunxi, uml, x86
    usbmultiselect    ¿, -, Yes, 1x 1.1, 1x 1.1 Device, 1x 2.0, 1x 2.0 Device, 2x 2.0, 3x 2.0, 4x 2.0, 5x 2.0, 6x 2.0, more than 6x 2.0, 1x 3.0, 2x 3.0, 3x 3.0, 4x 3.0, 5x 3.0, 6x 3.0, more than 6x 3.0, Mod, 1x Device, 1x Header, 1x OTG, 1x µUSB (charging), 1x µUSB (power), 1x Power only
    wlan24            ¿, -, b, b/g, b/g/n
    wlan50            ¿, -, a, a/n, a/n/ac
    yesno             ¿, Yes, No

Update datapages in the OpenWrt wiki

  1. Make sure to disable write access to <wikiroot>/toh/hwdata, for all users, also logged in ones. → admin / ACL
    (If write access is allowed, pageedits by users during the index updating process may confuse the database and lead to old data being displayed.)
  2. Delete the current data in <wikiroot>/toh/hwdata
  3. Cleanup database → admin / Data Plugin: Clean up database / Clean database (should return with big number of entries removed)
  4. Go through steps 1...4 as shown above. (Yes, after the update do a database cleanup once again: Should return with 0 removed entries)

Add / modify css for data plugin

  • for the default style definitions, see <wikihome>/lib/plugins/data/style.css
    • search for div.dataplugin_entry dl (should be right at the top) → change width: to 85% (or more), in order to provide more space for long urls
  • for user style definitions (e.g. - - - - dataentry techdata - - - -), see <wikihome>/conf/userstyle.css
    • Add to <wikihome>/conf/userstyle.css:
      div.techdata {
        /* code for the container block, such as position, layout and borders, goes here */
        /* width is further limited by / can not get bigger than the default value set in ''<wikihome>/lib/plugins/data/style.css'' */
        /*float: right;*/
        width: 100%;
      }
      
      div.techdata dl > dt {
        /* this will allow you to style the definition terms */
        font-family: sans-serif;
        width: 30%;
      }
      
      div.techdata dl > dt + dd {
        /* this will allow you to style to the definition values */
        font-family: sans-serif;
        width: 65%;
      }
      
      div.techdata dl dd.name{
        /* this makes the name larger than the rest */
        font-size: 80%;
      }

Add / modify Brands

  • admin / Data Plugin: Field Aliases
  • Line brand → Column Valid Values → add the new brand, sort it alphabetically (there's no automatism doing that for you)
  • Line brand-new = Line brand + additional Please select preceding the list of brands. It's a dirty workaround to get the Please select into the valid values, which is needed for creation of new dataentries (and only there; we don't want the Please select as option in the dataentries).
  • Save Aliases → New / modified brand is now available.

Community

  1. Create new ToH page using "Structured data plugin" syntax (that would be 4 lines) → Done, several examples available in demowiki
  2. Create new "dataentry" pages (one for each device) → Done, see https://dl.dropboxusercontent.com/u/9061436/openwrt/toh/latest/wikitxt.tgz
  3. Add template page for new dataentries → Done, available in demowiki (to be refined and polished)
  4. Add "permanent note" page for dataentries, including warning regarding testing-phase (automatically included in the datapages) → Done, available in demowiki
  5. Create "click here for new dataentry page" page (bureaucracy) → Done
  6. Create "click here for new device page" page (bureaucracy) → Done
  7. Rework template_device, in order to make use of the data in the dataentries → WIP

Database size and performance

Troubleshooting

Double dataentries

Issue opened on github: Double pid for same dataentry #187

Problem:

  • Double dataentries are shown in datatables
  • Double pageid / pageid not matching the set pageid caption
  • Editing the respective dataentry and saving it again doesn't help
  • Cleaning up the database doesn't help
  • Searchindex update doesn't help
  • Searchindex rebuild doesn't help

Root cause: No clear indication of the root cause for this behaviour.

  • Guess #1: Pagemoves by pagemove plugin and/or deletion of dataentry pages corrupts the sqlite database.
  • Guess #2: php caching (apc, apcu, Zend OPcache)

Solution #1: Manually edit the sqlite database via admin > SQLite Interface

There are two ways to find and fix the problematic dataentries in the sqlite db:

  1. Find possible doubles via double page / double title (Mind that not all findings are necessarily doubles!)
    1. via double 'page' → doesn't work, since page names of doubles are in fact different (double has additional X'0A' at the end)
    2. via double 'title'
      SELECT *
      FROM pages
      WHERE title IN (
      SELECT title
      FROM pages
      GROUP BY title
      HAVING (COUNT(title ) > 1)
      )
      ORDER BY title
    3. Manually check for presence of extra character behind pagename by simply marking the pagename (like for copy&paste).
      • If there's no extra character present, the page is not affected.
      • If extra character present, note the affected pid, e,g, 1129, 1130, >1134
    4. Manually delete the data with the double pid from the data and the pages tables.
      DELETE FROM data WHERE pid='1129' OR pid='1130' OR pid>'1134'
      DELETE FROM pages WHERE pid='1129' OR pid='1130' OR pid>'1134'
  2. via extra '0A' (=CHAR(10)=LF=\n) at the end of the pagename (preferred method)
    /* First find affected pages */
    SELECT * FROM data WHERE pid IN (SELECT pid FROM pages WHERE page LIKE '%' || X'0A' || '%');
    SELECT * FROM pages WHERE page LIKE '%' || X'0A' || '%';
    /* Then delete those entries */
    DELETE FROM data WHERE pid IN (SELECT pid FROM pages WHERE page LIKE '%' || X'0A' || '%');
    DELETE FROM pages WHERE page LIKE '%' || X'0A' || '%';

Database should be clean again now. Check with your favourite datatable, e.g. toh_minimal_all

Solution #2 (only few pages affected):

  1. Delete the respective dataentry page(s)
  2. Clean the database (admin > Data plugin: Clean database)
  3. Restore the previously deleted dataentry page(s)
  4. Check datatable → should be clean now, showing no more doubles.

Solution #3 (many pages affected): Delete database, rebuild index - Delete data plugin database <wikihome>/data/meta/data.sqlite
Beware: This also deletes the type alias definitions → need to be re-entered after the deletion. - Deactivate hidepage plugin via admin > Hidepages → check hidepages»ignorepattern → save config - Rebuild searchindex → admin > Searchindex manager > Rebuild Index - Reactivate hidepage plugin via admin > Hidepages → uncheck hidepages»ignorepattern → save config

Solution #3 is no real solution, since deletion of the complete database also deletes the alias definitions. You do not want this,since the manual rebuild of the alias definitions is quite some work.

wiki/installation_of_the_new_toh_functionality.txt · Last modified: 2016/07/25 19:10 by tmomas