Vagrant
This is the setup for a Vagrant instance that can be used for testing and development of the extension.
Initial setup
The usual setup is as described on mw:MediaWiki-Vagrant, and yes you need Git
mkdir ~/picklespace cd ~/picklespace git clone --recursive https://gerrit.wikimedia.org/r/mediawiki/vagrant . ./setup.sh vagrant roles enable pickle vagrant up # first “up” will do a provision
The first vagrant up will provision and start the instance.
Additional tweaks for the guest
Locales
If you get complaints about missing locales, usually from perl, then the problem is most likely missing locales in the vagrant box. Tunnel into the box with vagrant ssh and first check if /etc/ssh/sshd_config has a line AcceptEnv LANG LC_*, if it is commented then uncomment the line. Then do
sudo locale-gen nb_NO
sudo locale-gen nb_NO.UTF-8
Replace nb_NO with whatever locale that works for you. Then do
sudo dpkg-reconfigure locales
sudo update-locale LANG=nb_NO.UTF-8
Update & Upgrade
Make sure everything is updated
vagrant ssh sudo apt-get update && sudo apt-get upgrade
Now reload the vagrant instance
exit vagrant reload
Git update
At this point you should do a vagrant git-update in the host directory, otherwise you will probably run into problems later. Then do a vagrant reload to make sure everything works.
At this point you should clean up the client for outdated apt packages. Do a
vagrant ssh sudo apt-get autoremove exit
Note that this still leaves the box running old software, and you should do a sudo apt-get update && sudo apt-get upgrade from time to time.
Resource allocations
Vagrant resources
It could be interesting to adjust the amount of memory and number of CPU cores used during testing. Such resource allocations are described on mw:MediaWiki-Vagrant#Adjust the resources allocated to the VM? Especially note the code snippet for ~/picklespace/Vagrantfile-extra.rb.
Vagrant.configure('2') do |config| config.vm.provider :virtualbox do |vb| # See http://www.virtualbox.org/manual/ch08.html for additional options. vb.customize ['modifyvm', :id, '--memory', '1536'] vb.customize ['modifyvm', :id, '--cpus', '2'] end end
To restart the instance do vagrant reload.
If the memory is too low, then the test run in guest will end in a fork failed. When that happen, increase the memory until the test runs ok. With fastest it should be sufficient to set this around “1024”, and “1536” is enough. If it is set to high, then it seems like the test runs take more time. This could be due to garbage collection.
If the number of CPUs are increased beyond the actual number of cores, an increase in run time might be observed. When that happen, decrease the number of CPUs available to the vagrant instance.
Composer resources
There can be timeouts in the Vagrant guest instance during composer runs. To adjust for the increased time do something like
vagrant ssh
composer config --global process-timeout 900
Linting
Guest linting
Install additional Lua stuff
vagrant ssh
sudo apt-get install lua5.1
sudo apt-get install luarocks
sudo apt-get install unzip
sudo luarocks install luacheck
TODO: Probably be fixed in a few days. (T224791)
Seems like these two must be global to work properly
sudo apt install pdepend phpmd
The previous gives old versions, but they sort of work.
Most of the linting can be installed by the following (or vagrant git-update)
cd /vagrant/mediawiki/extensions/Pickle composer install
Append a line PATH="$PATH:/vagrant/mediawiki" to the file ~/.profile, do an exit and vagrant ssh to get back into the box.
Given that a standard Mediawiki-Vagrant is used then sudo must be used while calling npm. First update npm itself
cd /vagrant/mediawiki/extensions/Pickle sudo npm update -g npm
then install all dependencies
sudo npm install
Make sure we have a few additions
sudo npm install -g grunt-cli --save-dev npm install grunt --save-dev
(Those are only used for linting in the guest!)
Update nano ./node_modules/grunt-contrib-lualint/tasks/lualint.js, change ./libs/lualint to luacheck. The check done by lualint is rather rudimentary compared to luacheck, but the later has no grunt-module.
If you run grunt --force at this point you should get failures involving lualint, and luac.
Linting with PHP_CodeSniffer require (skip?)
sudo apt-get install php-pear sudo pear channel-update pear.php.net sudo pear install PHP_CodeSniffer
TODO: Got a lot of undefined here. Outdated repo? (Bug #21218)
Host linting
Some needs to be set up in the host. Usually the editing software needs additional tweaking, such as if you are using w:Visual Studio Code. The following assumes you are on a host running Ubuntu 16.10
Part of this is from PHPraxis – PHP and more: Install PHP 5.6 (or 5.5) in Ubuntu 16.04 LTS (Xenial Xerus) (Apache-packages might be skipped, possibly also databases.)
LC_ALL=C.UTF-8 sudo add-apt-repository ppa:ondrej/php sudo apt-get remove php-pear sudo apt-get -y install composer npm curl git sudo apt-get -y install php5.6 php5.6-xml php5.6-curl php5.6-json php5.6-mbstring php5.6-mysql php5.6-cli php5.6-sqlite3 php5.6-mcrypt sudo apt-get -y install php7.0 php7.0-xml sudo apt-get -y install libapache2-mod-php5.6 libapache2-mod-php7.0 mysql-server-5.7 apache2 sudo apt-get install php-codesniffer php-gettext php-pear php-php-gettext php-xdebug php-pear npm install composer install sudo apt-get update && sudo apt-get upgrade
To shift from php 5.6 to php 7.0
sudo a2dismod php5.6 ; sudo a2enmod php7.0 ; sudo service apache2 restart
To shift from php 7.0 to php 5.6
sudo a2dismod php7.0 ; sudo a2enmod php5.6 ; sudo service apache2 restart
Assuming you are on a Ubuntu machine, set up luarocks
sudo apt-get install luarocks
then set up luacheck
sudo luarocks install luacheck
Mess detection
A lot of problems. Seems like the host works as advertised for pdepend and phpmd, while the guest has problems with newer versions. For now the only versions that works are the pretty old
composer global require pdepend/pdepend:1.* phpmd/phpmd:1.*
Manual testing
There are a number of code pages in a file pages.xml. This file can be imported by composer import, and if necessary a new file built by composer export. The core pages should be available from Main Page, but note especially the page VagrantRole/Pickle/testcases.
Verification of the setup
To do some very simple manual testing, add the following on a page Module:Test in the Vagrant-instance. It should be available at http://127.0.0.1:8080/wiki/Module:Test (HHVM) or at http://php5.local.wmftest.net:8080/wiki/Module:Test (PHP5)
local h = {} h.foo = function() return 'bar' end return h
At this point the module should have a category “Modules with unknown tests”, which is clearly wrong as there are no tests! This is a bug that needs fixing, it is a default handler that doesn't check if the page exists.
Then add the following on a page Module:Test/pickle
local h = {} function h.tap() return "1..1\nok 1" end return h
The h.tap() fakes a test report, as the code for reporting test states isn't finalized yet.
Page indicators and tracking categories should now be set, and logging events generated on state changes. The page Module:Test/pickle should have a page indicator saying “Good” in some language, and after reloading Module:Test that page should be “Good” too.
Verification of development
The development starts to come together, and it is possible to verify the code through the development console. In the present version the library is bootstrapped through a side effect of accessing describe(), which also double as the outer test case (or example) in a spec. The “loading” is requiring a number of libs, registering some functions to access those libraries, and then doing the usual function associated with describe. The resulting model can be observed in the lua console.
= mw.dumpObject( p ) -- this is a table
Note that the console does not handle comments, and those must be left out when you do copy-pasting.
The initial result, the value set to p, is the unevaluated model. To get to the (TODO)
A very small “shim” is available as part of the loading of non-pure lua libs. This is located at mw.pickle, and contains configuration data used during bootstrapping. Before bootstrap only configuration data and the function to do bootstrapping is available.
= mw.dumpObject( mw.pickle ) -- this is a table = mw.dumpObject( describe ) -- this is a function
Actual content of the structure may change.
After bootstrapping the global environment will contain numerous other functions, and possibly some private tables. One private table is _reports and will expand and contract as tests are run. To some degree this can be safely manipulated through use of spies. The bootstrapping happens in the normal running code, and not in the console. What you inspect in the console is the observable model without the additional functions that will be installed through the bootstrap process. That means you will see the describe function, but you will not see the context or it functions unless you run describe inside the console.
= mw.dumpObject( _reports ) -- should return 'nil' describe():eval() -- this fill '_reports' with a few lines = mw.dumpObject( _reports ) -- should now return a table
Automated testing
Testing of PHP files are mainly done from composer, which also include access points to test entries for Lua files. Testing of Javascript files and other files are done from grunt. Check out composer.json and Gruntfile.js.
Unit tests
To run the unit tests on the code base start the vagrant instance
vagrant up vagrant ssh
and then run the test for phpunit
cd /vagrant/mediawiki
sudo -u www-data php5 tests/phpunit/phpunit.php --wiki wiki --group Pickle
or run the style tests and unit tests for composer
cd /vagrant/mediawiki/extensions/Pickle composer test composer unit
Running the unit tests through composer makes the tests run at twice the speed.
Bugs & problems
Role “scribunto”
Until the scribunto role is fixed, there are random critical errors during testing, the role must be manually set to use standalone instead of sandbox
# Class: role::scribunto # Configures Scribunto, an extension for embedding scripting languages # in MediaWiki. class role::scribunto { include ::role::codeeditor include ::role::syntaxhighlight mediawiki::extension { 'Scribunto': settings => [ '$wgScribuntoEngineConf["luastandalone"]["luaPath"] = "/vagrant/mediawiki/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua"', '$wgScribuntoDefaultEngine = "luastandalone"', '$wgScribuntoUseGeSHi = true', '$wgScribuntoUseCodeEditor = true', ], notify => Service['apache2'], require => [ Mediawiki::Extension['CodeEditor'], Mediawiki::Extension['SyntaxHighlight_GeSHi'], ], } }
Failed dependencies
During first vagrant up I often gets missing or failed dependencies. Usually a vagrant reload --provision solve the problem.
Missing /var/log
Sometimes I'm stuck on missing /var/log on guest. Usually I just do a vagrant destroy and start over, but sometime I'm stuck. Only thing that worked for me in those cases was a vagrant box remove trusty-cloud. This is a very hard reset because something has gone really wrong in the guest.
RPC failed
Sometime I get error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. In those cases start from scratch and hope the best. It seems like the error is triggered by packet collisions, then gnuTLS fails, and then curl can't restart. See also Phabricator T152801 and Launchpad: bug#1111882.
Timeout after updating the key
Usually something like “Timed out while waiting for the machine to boot.” Try a new vagrant reload, usually it will work after the restart, or slap the box with vagrant reload --provision.
Sometimes it helps to do a ACPI shutdown, or restart, and then an ACPI shutdown on the instance. Also, consider a restart on the host to release stuck bound sockets.
Timeout on waiting for NFS
Usually this is a failed restart of NFS on host. Do a vagrant halt, and restart NFS (hard), or reboot the host (easy). If NFS fails, it could be an indication that other things is stuck too.
NFS is tricky to restart, read the manual. And yes, it is not unlikely NFS has failed on the guest too.
Not sure about this: It seems like updates of VirtualBox often leaves an old version of the Guest Additions. That makes the next vagrant up fail. It seems like a vagrant destroy is necessary to make the guest work again, so all local additions must be reapplied. Because other guests might work as usual this can be very confusing.
Ports in use
Sometimes I'm stuck with ports in use on host or guest. Usually it is the host, and usually it is use of port 8080. Seems like only thing that surely solves that is a vagrant halt, which should say that the instance is not running, reboot and then vagrant up --provision.
It could be other processes that use the ports, and if those starts again after reboot you're stuck. Try to track them down with netstat -na | grep 8080 (or whatever offending port) and kill them, or reassign the ports. If it is port 8080 on the host then the port is set in the file settings.d/wikis/wgConf.php and the config settings.d/wikis/wiki/wgConf.php.
Symfony
Unless the provisioning is fixed (symfony/process dependency in SyntaxHighlight not loaded) there will be a lot of red error messages at this point if you skipped vagrant git-update, or there will be complaints about missing classes when you open the main page for you mediawiki instance. The following should preferably be done before setting up any roles.
Use a vagrant git-update and possibly a vagrant reload --provision to make the complaint go away. If they still don't go away try to use vagrant ssh to tunnel into the vagrant box and then cd /vagrant/mediawiki/extensions and for each extension cd into the directory and do a composer update. Afterward, do a vagrant reload --provision.
Test does not use all cores
Composer tries to get fastest, but unless composer update is run, that is vagrant git-update, it isn't available. It isn't that much that can be run in parallel on Mediawiki, so the tests are slow anyhow. With fastest I get a 20% drop in overall run time for the tests. The ordinary Lua tests run from PHP are awfully slow.
Notes
Running boxes can be listed with
vagrant global-status
Abandoned boxes can be pruned with
vagrant global-status --prune