From 46838813cd53e58112db5f6524cb94db46bdceb6 Mon Sep 17 00:00:00 2001 From: MCUdude Date: Sun, 4 Oct 2020 19:36:20 +0200 Subject: [PATCH] Update arduino-ci-script to version 1.3.3 --- .../arduino-ci-script/CONTRIBUTING.md | 24 - .../arduino-ci-script/ISSUE_TEMPLATE.md | 1 - avr/travis-ci/arduino-ci-script/LICENSE | 2 +- .../PULL_REQUEST_TEMPLATE.md | 1 - avr/travis-ci/arduino-ci-script/README.md | 203 +- .../arduino-ci-script/arduino-ci-script.sh | 1759 ++++++++++++++--- .../etc/astyle-configurations/1.conf | 33 + .../etc/astyle-configurations/2.conf | 46 + .../etc/astyle-configurations/3.conf | 125 ++ .../arduino-ci-script/etc/autoformat.sh | 1 + .../etc/codespell-ignore-words-list.txt | 4 + 11 files changed, 1862 insertions(+), 337 deletions(-) delete mode 100644 avr/travis-ci/arduino-ci-script/CONTRIBUTING.md delete mode 100644 avr/travis-ci/arduino-ci-script/ISSUE_TEMPLATE.md mode change 100644 => 100755 avr/travis-ci/arduino-ci-script/LICENSE delete mode 100644 avr/travis-ci/arduino-ci-script/PULL_REQUEST_TEMPLATE.md mode change 100644 => 100755 avr/travis-ci/arduino-ci-script/README.md mode change 100644 => 100755 avr/travis-ci/arduino-ci-script/arduino-ci-script.sh create mode 100755 avr/travis-ci/arduino-ci-script/etc/astyle-configurations/1.conf create mode 100755 avr/travis-ci/arduino-ci-script/etc/astyle-configurations/2.conf create mode 100755 avr/travis-ci/arduino-ci-script/etc/astyle-configurations/3.conf create mode 100755 avr/travis-ci/arduino-ci-script/etc/autoformat.sh create mode 100755 avr/travis-ci/arduino-ci-script/etc/codespell-ignore-words-list.txt diff --git a/avr/travis-ci/arduino-ci-script/CONTRIBUTING.md b/avr/travis-ci/arduino-ci-script/CONTRIBUTING.md deleted file mode 100644 index f17d9825..00000000 --- a/avr/travis-ci/arduino-ci-script/CONTRIBUTING.md +++ /dev/null @@ -1,24 +0,0 @@ -# Contribution Rules -Thanks for your interest in contributing to this free open source project! Please take the time to read and follow these rules before submitting an issue report or pull request. - -## Issues -- Do you need help using this project? Support requests should be made to the appropriate section of the [Arduino forum](http://forum.arduino.cc) rather than an issue report. Feel free to [send me a PM](http://forum.arduino.cc/index.php?action=pm;sa=send;u=224903) with a link to your forum thread. **Support will not be provided via PM**, I prefer to help you publicly so that others with the same question may benefit from the information. **Issue reports are to be used to report bugs or make feature requests only.** -- Before submitting a bug report test using the [latest version of the project](https://github.com/per1234/arduino-ci-script/archive/master.zip) to be sure it hasn't already been fixed. **Don't report issues that only occur with old versions of the project.** -- Search [existing pull requests and issues](https://github.com/per1234/arduino-ci-script/issues?q=) to be sure it hasn't already been reported. **Do not submit duplicate issue reports.** If you have additional information to provide about the issue then please comment on that issue. -- Open an issue at https://github.com/per1234/arduino-ci-script/issues/new. -- Describe the issue and what behavior you were expecting. Post complete error messages using [markdown code fencing](https://guides.github.com/features/mastering-markdown/#examples). -- Provide a full set of steps necessary to reproduce the issue. Demonstration code should be complete, correct and simplified to the minimum amount of code necessary to reproduce the issue. Please use [markdown code fencing](https://guides.github.com/features/mastering-markdown/#examples) when posting code. -- Be responsive. I may need you to provide more information, please respond as soon as possible. -- If you find a solution to your problem update your issue report with an explanation of how you were able to fix it and close the issue. - -## Pull Requests -- Search [existing pull requests and issues](https://github.com/per1234/arduino-ci-script/pulls?q=) to make sure the change hasn't already been proposed. -- Comment your code. The focus of Arduino is learning so it's best to be a bit more thorough about documenting code. -- Follow the formatting conventions used throughout the rest of the project. Remove all trailing whitespace. -- If appropriate, add or update tests in the [.travis.yml file](https://github.com/per1234/arduino-ci-script/blob/master/.travis.yml). -- Update the [documentation](https://github.com/per1234/arduino-ci-script/blob/master/README.md) if your changes require it. This should be done in the same commit as the change. -- **All commits must be atomic**. This means that the commit completely accomplishes a single task. Each commit should result in fully functional code. Multiple tasks should not be combined in a single commit. For more information please read http://www.freshconsulting.com/atomic-commits. -- Commit messages: Use the [imperative mood](http://chris.beams.io/posts/git-commit/#imperative) in the commit title. Completely explain the purpose of the commit. Please read http://chris.beams.io/posts/git-commit for more tips on writing good commit messages. -- Each pull request should address a single bug fix or enhancement, this may consist of multiple commits. If you have multiple, unrelated fixes or enhancements to contribute, then do each in a separate pull request. -- Open a pull request at https://github.com/per1234/arduino-ci-script/compare. -- If your pull request fixes an issue in the issue tracker, use the [closes/fixes/resolves syntax](https://help.github.com/articles/closing-issues-via-commit-messages) in the body to denote this. diff --git a/avr/travis-ci/arduino-ci-script/ISSUE_TEMPLATE.md b/avr/travis-ci/arduino-ci-script/ISSUE_TEMPLATE.md deleted file mode 100644 index 3bdf3ca0..00000000 --- a/avr/travis-ci/arduino-ci-script/ISSUE_TEMPLATE.md +++ /dev/null @@ -1 +0,0 @@ -Read the Issues section of the Contribution Rules at the "guidelines for contributing" link above before submitting an issue report. diff --git a/avr/travis-ci/arduino-ci-script/LICENSE b/avr/travis-ci/arduino-ci-script/LICENSE old mode 100644 new mode 100755 index e35d9952..56c77ec3 --- a/avr/travis-ci/arduino-ci-script/LICENSE +++ b/avr/travis-ci/arduino-ci-script/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2017 +Copyright (c) 2020 per1234, Vitalii Tereshchuk Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/avr/travis-ci/arduino-ci-script/PULL_REQUEST_TEMPLATE.md b/avr/travis-ci/arduino-ci-script/PULL_REQUEST_TEMPLATE.md deleted file mode 100644 index 70b835d5..00000000 --- a/avr/travis-ci/arduino-ci-script/PULL_REQUEST_TEMPLATE.md +++ /dev/null @@ -1 +0,0 @@ -Read the Pull Requests section of the Contribution Rules at the "guidelines for contributing" link above before submitting a pull request. diff --git a/avr/travis-ci/arduino-ci-script/README.md b/avr/travis-ci/arduino-ci-script/README.md old mode 100644 new mode 100755 index 2b51dde9..09eb6dbd --- a/avr/travis-ci/arduino-ci-script/README.md +++ b/avr/travis-ci/arduino-ci-script/README.md @@ -1,58 +1,79 @@ arduino-ci-script ========== -Bash script for continuous integration of [Arduino](http://www.arduino.cc/) projects. I'm using this centrally managed script for multiple repositories to make updates easy. I'm using this with [Travis CI](http://travis-ci.org/) but it could be easily adapted to other purposes. +Bash script for continuous integration of [Arduino](http://www.arduino.cc/) projects. This is currently targeted for use with [Travis CI](http://travis-ci.org/) but it could be easily adapted to other purposes. [![Build Status](https://travis-ci.org/per1234/arduino-ci-script.svg?branch=master)](https://travis-ci.org/per1234/arduino-ci-script) -#### Installation -- You can download a .zip of all the files from https://github.com/per1234/arduino-ci-script/archive/master.zip -- Include the script in your project by adding the following line: -```bash - - source arduino-ci-script.sh -``` -- It's possible to leave the script hosted at this repository. -```bash - - source <(curl -SLs https://raw.githubusercontent.com/per1234/arduino-ci-script/master/arduino-ci-script.sh) + +### Table of contents +- [Installation](#installation) +- [Usage](#usage) +- [Publishing job reports](#publishing-job-reports) +- [Troubleshooting](#troubleshooting) +- [Contributing](#contributing) + + +### Installation +The script can be used in multiple ways: + +#### Clone the latest release +Include the latest release of the script in your project by adding the following lines to your build configuration file: +```yaml + # Clone the script repository + - git clone --depth 1 https://github.com/per1234/arduino-ci-script.git "${HOME}/scripts" + - cd "${HOME}/scripts" + # Get new tags from the remote + - git fetch --tags + # Checkout the latest tag + - git checkout $(git describe --tags `git rev-list --tags --max-count=1`) + - source "${HOME}/scripts/arduino-ci-script.sh" ``` -- The above doesn't allow you control over the version and would not be a good idea for security reasons if you use the functions that take a GitHub token argument. You could use the script at a specific point in the commit history with something like this: -```bash - - git clone https://github.com/per1234/arduino-ci-script.git "${HOME}/arduino-ci-script" - - cd "${HOME}/arduino-ci-script" - - git checkout 0355314d45970cd33e52ebba9967646a063ea9eb - - source "${HOME}/arduino-ci-script/arduino-ci-script.sh" + +#### Local copy +If you're passing a token to the script's publish report functions then best security practices would be to use a static copy of the script so you can be sure of the commands the token is used with: +- Download the latest version of the script from https://github.com/per1234/arduino-ci-script/releases by clicking one of the **Source code** links. +- Unzip the downloaded file. +- Copy arduino-ci-script.sh to a convenient location. +- Include the script in your project by adding the following line to your build configuration file: +```yaml + - source arduino-ci-script.sh ``` +Be sure to check for new releases of the script so that you can benefit from the ongoing development work. You can receive notifications of releases by [watching the repository](https://github.com/per1234/arduino-ci-script/subscription). + +### Usage +See https://github.com/per1234/WatchdogLog/blob/master/.travis.yml for an example of the script in use. + +Please configure your continuous integration system to make the minimum number of downloads and sketch verifications necessary to effectively test your code. This will prevent wasting Arduino and Travis CI's bandwidth and keep the build durations short. -#### Usage -See https://github.com/per1234/WatchdogLog/blob/master/.travis.yml for an example of the script in use. Please configure your continuous integration system to make the minimum number of downloads and sketch verifications necessary to effectively test your code. This will prevent wasting Arduino and Travis CI's bandwidth while making the builds run fast. ##### `set_script_verbosity SCRIPT_VERBOSITY_LEVEL` Control the level of verbosity of the script's output in the Travis CI log. Verbose output can be helpful for debugging but in normal usage it makes the log hard to read and may cause the log to exceed Travis CI's maximum log size of 4 MB, which causes the job to be terminated. The default verbosity level is `0`. - Parameter: **SCRIPT_VERBOSITY_LEVEL** - `0`, `1` or `2` (least to most verbosity). ##### `set_application_folder APPLICATION_FOLDER` -- Parameter: **APPLICATION_FOLDER** - The folder to install the Arduino IDE to. This should be set to `/usr/local/share` or a subfolder of that location. The folder will be created if it doesn't already exist. The Arduino IDE will be installed in the `arduino` subfolder. +- Parameter: **APPLICATION_FOLDER** - The folder to install the Arduino IDE (and Artistic Style if you use `check_code_formatting`) to. This should be set to `/usr/local/share` or a subfolder of that location. The folder will be created if it doesn't already exist. The Arduino IDE will be installed in the `arduino` subfolder. ##### `set_sketchbook_folder SKETCHBOOK_FOLDER` - Parameter: **SKETCHBOOK_FOLDER** - The folder to be set as the Arduino IDE's sketchbook folder. The folder will be created if it doesn't already exist. Libraries installed via `install_library` will be installed to the `libraries` subfolder. Non-Boards Manager hardware packages installed via `install_package` will be installed to the `hardware` subfolder. This setting is only supported by Arduino IDE 1.5.6 and newer. ##### `set_board_testing BOARD_TESTING` -Turn on/off checking for errors with the board definition that don't affect sketch verification such as missing bootloader file. If this is turned on and an error is detected the build will be failed. This feature is off by default. +Turn on/off checking for problems with the board definition that generate a warning message during sketch verification but don't ordinarily cause it to fail, such as missing bootloader file. If this is turned on and a problem is detected the `build_sketch` command will return a non-zero exit status. This feature is off by default. - Parameter: **BOARD_TESTING** - `true`/`false` ##### `set_library_testing LIBRARY_TESTING` -Turn on/off checking for errors with libraries that don't affect sketch verification such as missing or invalid items in the library.properties file. If this is turned on and an error is detected the build will be failed. This feature is off by default. +Turn on/off checking for problems with libraries that generate a warning message during sketch verification but don't ordinarily cause it to fail, such as missing or invalid items in the library.properties file. If this is turned on and a problem is detected the `build_sketch` command will return a non-zero exit status. This feature is off by default. - Parameter: **LIBRARY_TESTING** - `true`/`false` ##### Special version names: - - `all`: Refers to all versions of the Arduino IDE (including the hourly build). In the context of `install_ide` this means all IDE versions listed in the script (those that support the command line interface, 1.5.2 and newer). In the context of all other functions this means all IDE versions that were installed via `install_ide`. - - `oldest`: The oldest release version of the Arduino IDE. In the context of `install_ide` this is the oldest of the IDE versions listed in the script (1.5.2, the first version to have a command line interface). In the context of build_sketch this means the oldest IDE version that was installed via `install_ide`. - - `newest`: Refers to the newest release version of the Arduino IDE (not including the hourly build unless hourly is the only version on the list). In the context of `install_ide` this means the newest IDE version listed in the script. In the context of all other functions this means the newest IDE version that was installed via `install_ide`. + - `all`: Refers to all versions of the Arduino IDE (including the hourly build). In the context of `install_ide` this means all IDE versions compatible with the script (those that support the command line interface, 1.5.2 and newer). In the context of all other functions this means all IDE versions that were installed via `install_ide`. + - `oldest`: The oldest release version of the Arduino IDE. In the context of `install_ide` this is the oldest of the IDE versions compatible with the script (1.5.2, the first version to have a command line interface). In the context of build_sketch this means the oldest IDE version that was installed via `install_ide`. + - `newest`: In the context of `install_ide` this means the newest IDE release version. In the context of all other functions this means the newest IDE release version that was installed via `install_ide`. 'newest' will only match to the hourly build if that is the only version available. - `hourly`: The hourly build of the Arduino IDE. Note that this IDE version is intended for beta testing only. ##### `install_ide [IDEversionList]` -Install a list of version(s) of the Arduino IDE. -- Parameter(optional): **IDEversionList** - A list of the versions of the Arduino IDE you want installed, in order from oldest to newest. e.g. `'("1.6.5-r5" "1.6.9" "1.8.2")'`. If no arguments are supplied all IDE versions will be installed. I have defined all versions of the Arduino IDE that have a command line interface in the script for the sake of being complete but I really don't see much reason for testing with the 1.5.x versions of the Arduino IDE. Please only install the IDE versions you actually need for your test to avoid wasting Arduino's bandwidth. This will also result in the builds running faster. Installation of the IDE will be skipped if it's found to already be installed in the folder specified via the `set_application_folder` function so `install_ide` can also be used simply to inform the script which IDE versions are available. +Install a list of Arduino IDE version(s). +- Parameter(optional): **IDEversionList** - A list of the versions of the Arduino IDE you want installed, in order from oldest to newest. e.g., `'("1.6.5-r5" "1.6.9" "1.8.2")'`. If no arguments are supplied all IDE versions will be installed. The script allows you to install all IDE versions with a command line interface (1.5.2 and newer) for the sake of being complete but I don't see a good reason for testing with the 1.5.x versions of the Arduino IDE. Please only install the IDE versions you actually need for your test to avoid wasting Arduino's bandwidth. This will also result in a shorter build duration. Installation of the IDE will be skipped if it's found to already be installed in the folder specified via the `set_application_folder` function so `install_ide` can also be used simply to inform the script which IDE versions are available. ##### `install_ide startIDEversion [endIDEversion]` Install a range of version(s) of the Arduino IDE. @@ -60,69 +81,101 @@ Install a range of version(s) of the Arduino IDE. - Parameter(optional): **endIDEversion** - The newest version of the Arduino IDE to install. If this argument is omitted then only startIDEversion will be installed. ##### `install_package` -"Manually" install the hardware package from the current repository. Packages are installed to `$SKETCHBOOK_FOLDER/hardware. Assumes the hardware package is located in the root of the download or repository and has the correct folder structure. +"Manually" install the hardware package from the current repository. Packages are installed to `$SKETCHBOOK_FOLDER/hardware`. Assumes the hardware package is located in the root of the download or repository and has the correct folder structure. ##### `install_package packageURL` -"Manually" install a hardware package downloaded as a compressed file. Packages are installed to `$SKETCHBOOK_FOLDER/hardware. Assumes the hardware package is located in the root of the file and has the correct folder structure. -- Parameter: **packageURL** - The URL of the hardware package download. The protocol component of the URL (e.g. `http://`, `https://`) is required. +"Manually" install a hardware package downloaded as a compressed file. Packages are installed to `$SKETCHBOOK_FOLDER/hardware`. Assumes the hardware package is located in the root of the file and has the correct folder structure. +- Parameter: **packageURL** - The URL of the hardware package download. The scheme component of the URL (e.g., `http://`, `https://`) is required. ##### `install_package packageURL [branchName]` -"Manually" install a hardware package. Packages are installed to `$SKETCHBOOK_FOLDER/hardware. Assumes the hardware package is located in the root of the repository and has the correct folder structure. -- Parameter: **packageURL** - The URL of the Git repository. The protocol component of the URL (e.g. `http://`, `https://`) is required. +"Manually" install a hardware package by cloning from a Git repository. Packages are installed to `$SKETCHBOOK_FOLDER/hardware`. Assumes the hardware package is located in the root of the repository and has the correct folder structure. +- Parameter: **packageURL** - The URL of the Git repository. The scheme component of the URL (e.g., `http://`, `https://`) is required. The URL must end in `.git`. - Parameter(optional): **branchName** - Branch of the repository to install. If this argument is not specified or is left blank the default branch will be used. ##### `install_package packageID [packageURL]` -Install a hardware package using the Arduino IDE (Boards Manager). Only the **Arduino AVR Boards** package is included with the Arduino IDE installation. Packages are installed to `$HOME/.arduino15/packages. You must call `install_ide` before this function. This feature is only available with Arduino IDE 1.6.4 and newer. -- Parameter: **packageID** - `package name:platform architecture[:version]`. If `version` is omitted the most recent version will be installed. e.g. `arduino:samd` will install the most recent version of **Arduino SAM Boards**. -- Parameter(optional): **packageURL** - The URL of the Boards Manager JSON file for 3rd party hardware packages. This can be omitted for hardware packages that are included in the official Arduino JSON file (e.g. Arduino SAM Boards, Arduino SAMD Boards, Intel Curie Boards). +Install a hardware package using the Arduino IDE (Boards Manager). Only the **Arduino AVR Boards** package is included with the Arduino IDE installation. Packages are installed to `$HOME/.arduino15/packages`. You must call `install_ide` before this function. This feature is only available with Arduino IDE 1.6.4 and newer. +- Parameter: **packageID** - `package name:platform architecture[:version]`. If `version` is omitted the most recent version will be installed. e.g., `arduino:samd` will install the most recent version of **Arduino SAMD Boards**. +- Parameter(optional): **packageURL** - The URL of the Boards Manager JSON file for 3rd party hardware packages. This can be omitted for hardware packages that are included in the official Arduino JSON file (e.g., Arduino SAM Boards, Arduino SAMD Boards, Intel Curie Boards). ##### `install_library` Install the library from the current repository. Assumes the library is in the root of the repository. The library is installed to the `libraries` subfolder of the sketchbook folder. ##### `install_library libraryName` Install a library that is listed in the Arduino Library Manager index. The library is installed to the `libraries` subfolder of the sketchbook folder. You must call `install_ide` before this function. This feature is only available with Arduino IDE 1.6.4 and newer installed. -- Parameter: **libraryName** - The name of the library to install. You can specify a version separated from the name by a colon, e.g. "LiquidCrystal I2C:1.1.2". If no version is specified the most recent version will be installed. You can also specify comma-separated lists of library names. +- Parameter: **libraryName** - The name of the library to install. You can specify a version separated from the name by a colon, e.g., "LiquidCrystal I2C:1.1.2". If no version is specified the most recent version will be installed. You can also specify comma-separated lists of library names. ##### `install_library libraryURL [newFolderName]` Download a library in a compressed file from a URL. The library is installed to the `libraries` subfolder of the sketchbook folder. -- Parameter: **libraryURL** - The URL of the library download or library name in the Arduino Library Manager. The protocol component of the URL (e.g. `http://`, `https://`) is required. This can be any compressed file format. Assumes the library is located in the root of the file. -- Parameter(optional): **newFolderName** - Folder name to rename the installed library folder to. This can be useful if the default folder name of the downloaded file is problematic. The Arduino IDE gives include file preference when the filename matches the library folder name. GitHub's "Download ZIP" file is given the folder name {repository name}-{branch name}. Library folder names that contain `-` or `.` are not compatible with Arduino IDE 1.5.6 and older, arduino will hang if it's started with a library using an invalid folder name installed. +- Parameter: **libraryURL** - The URL of the library download or library name in the Arduino Library Manager. The scheme component of the URL (e.g., `http://`, `https://`) is required. The download file can be in any compressed file format. Assumes the library is located in the root of the file. +- Parameter(optional): **newFolderName** - Folder name to rename the installed library folder to. This can be useful if the default folder name of the downloaded file is problematic. The Arduino IDE gives include file preference when the filename matches the library folder name. GitHub's "Download ZIP" file is given the folder name `{repository name}-{branch name}`. Library folder names that contain `-` or `.` are not compatible with Arduino IDE 1.5.6 and older, arduino will hang if it's started with a library using an invalid folder name installed. ##### `install_library libraryURL [branchName [newFolderName]]` -Install a library by cloning a Git repository). The library is installed to the `libraries` subfolder of the sketchbook folder. -- Parameter: **libraryURL** - The URL of the library download or library name in the Arduino Library Manager. The protocol component of the URL (e.g. `http://`, `https://`) is required. Assumes the library is located in the root of the repository. +Install a library by cloning a Git repository. The library is installed to the `libraries` subfolder of the sketchbook folder. Assumes the library is located in the root of the repository. +- Parameter: **libraryURL** - The URL of the library download or library name in the Arduino Library Manager. The scheme component of the URL (e.g., `http://`, `https://`) is required. The URL must end in `.git`. - Parameter(optional): **branchName** - Branch of the repository to install. If this argument is not specified or is left blank the default branch will be used. - Parameter(optional): **newFolderName** - Folder name to rename the installed library folder to. This can be useful if the default folder name of the downloaded file is problematic. The Arduino IDE gives include file preference when the filename matches the library folder name. Library folder names that contain `-` or `.` are not compatible with Arduino IDE 1.5.6 and older, arduino will hang if it's started with a library using an invalid folder name installed. If the `newFolderName` argument is specified the `branchName` argument must also be specified. If you don't want to specify a branch then use `""` for the `branchName` argument. ##### `set_verbose_output_during_compilation verboseOutputDuringCompilation` -Turn on/off arduino verbose output during compilation. This will show all the commands arduino runs during the process rather than just the compiler output. This is usually not very useful output and only clutters up the log. This feature is off by default. +Turn on/off `arduino` verbose output during compilation (same as the IDE's **File > Preferences > Show verbose output during: > compilation**). This will show all the commands `arduino` runs during the process rather than just the compiler output. This is usually not very useful output and only clutters up the log. This feature is off by default. - Parameter: **verboseOutputDuringCompilation** - `true`/`false` +##### `check_sketch_structure searchPath` +Check sketches to ensure they have the correct structure. +- Parameter: **searchPath** - Path containing sketches. The path will be searched recursively and all sketches found under it will be checked. + +##### `check_library_structure basePath [depth]` +Check libraries to ensure they have the correct structure. This will also run `check_sketch_structure` on all sketches bundled with the library. +- Parameter: **basePath** - Path containing a library. +- Parameter(optional): **depth** - Folder depth relative to `basePath` where the libraries are located. A depth of 0 will check the library located at `basePath`. The default value is 0. + +##### `check_library_properties searchPath [maximumSearchDepth]` +Check [library.properties](https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format) library metadata files for errors. +- Parameter: **searchPath** - Path containing library.properties. +- Parameter(optional): **maximumSearchDepth** - The recursive search depth. A depth of 0 will only search `searchPath` and no subfolders. The default value is 0. + +##### `check_keywords_txt searchPath [maximumSearchDepth]` +Check [keywords.txt](https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#keywords) files for errors. +- Parameter: **searchPath** - Path containing keywords.txt files. +- Parameter(optional): **maximumSearchDepth** - The recursive search depth. A depth of 0 will only search `searchPath` and no subfolders. The default value is 0. + +##### `check_library_manager_compliance libraryPath` +Make some additional checks for compliance with the requirements for adding a library to the [Library Manager index](https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ). This function should be used in combination with `check_library_structure` and `check_library_properties` to ensure full compliance with the requirements. +- Parameter: **libraryPath** - Path of the library to check. + +##### `check_code_formatting strictness excludedPathList targetPath` +Check code formatting for compliance with the Arduino code style. The [Artistic Style](http://astyle.sourceforge.net) formatter tool is used for this check. If it's not already installed, it will be installed to the `astyle` subfolder of the folder specified to `set_application_folder`. Note that in the Travis CI job logs, the `check_code_formatting` output is "folded" to make it easier to browse. You can click the triangle in the left margin of the command to unfold the output. +- Parameter: **strictness** - Determines how strict to be about code formatting compliance: 1 (least strict) - 3 (most strict). Each strictness level is based on the previous one, but with additional requirements. + - `1`: The Arduino IDE's auto format configuration. + - `2`: The configuration Arduino uses to format their example sketches. + - `3`: A custom configuration based on a study of the prevailing styles used in official Arduino code. +- Parameter: **excludedPathList** - A comma-separated list of paths to exclude from the check. +- Parameter: **targetPath** - The path to run the check on. All code files will be checked recursively. + ##### `build_sketch sketchPath boardID allowFail IDEversion` ##### `build_sketch sketchPath boardID allowFail [IDEversionList]` ##### `build_sketch sketchPath boardID allowFail startIDEversion endIDEversion` -Pass some parameters from .travis.yml to the script. `build_sketch` will echo the arduino exit status to the log, which is documented at https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#exit-status. +Verify/compile sketch(es). `build_sketch` will echo the `arduino` exit status to the log, which is documented at https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#exit-status. Note that in the Travis CI job logs, the compilation output is "folded" to make it easier to browse. You can click the triangles in the left margin to unfold. - Parameter: **sketchPath** - Path to a sketch or folder containing sketches. If a folder is specified it will be recursively searched and all sketches will be verified. -- Parameter: **boardID** - `package:arch:board[:parameters]` ID of the board to be compiled for. e.g. `arduino:avr:uno`. Board-specific parameters are only supported by Arduino IDE 1.5.5 and newer. +- Parameter: **boardID** - `package:arch:board[:parameters]` ID of the board to be compiled for. e.g., `arduino:avr:uno`. Board-specific parameters are only supported by Arduino IDE 1.5.5 and newer. - Parameter: **allowFail** - `true`, `require`, or `false`. Allow the verification to fail without causing the CI build to fail. `require` will cause the build to fail if the sketch verification doesn't fail. - Parameter: **IDEversion** - A single version of the Arduino IDE to use to verify the sketch. -- Parameter(optional): **IDEversionList** - A list of versions of the Arduino IDE to use to verify the sketch. e.g. `'("1.6.5-r5" "1.6.9" "1.8.2")'`. If no version list is provided all installed IDE versions will be used. +- Parameter(optional): **IDEversionList** - A list of versions of the Arduino IDE to use to verify the sketch. e.g., `'("1.6.5-r5" "1.6.9" "1.8.2")'`. If no version list is provided all installed IDE versions will be used. - Parameter: **startIDEversion** - The start (inclusive) of a range of versions of the Arduino IDE to use to verify the sketch. - Parameter: **endIDEversion** - The end (inclusive) of a range of versions of the Arduino IDE to use to verify the sketch. ##### `display_report` -Echo a tab separated report of all verification results to the log. The report is located in `${HOME}/arduino-ci-script_report`. Note that Travis CI runs each build of the job in a separate virtual machine so if you have multiple jobs you will have multiple reports. The only way I have found to generate a single report for all tests is to run them as a single job. This means not setting multiple matrix environment variables in the `env` array. See https://docs.travis-ci.com/user/environment-variables. The report consists of: -- Build timestamp +Echo a tab separated report of all verification results to the log. The report is located in the `${HOME}/arduino-ci-script_report` folder and will be named according to the build number and job number. Note that Travis CI runs each build of the job in a separate virtual machine so if you have multiple jobs you will have multiple reports. The only way I have found to generate a single report for all tests is to run them as a single job. This means not setting multiple matrix environment variables in .travis.yml's `env` array. See: https://docs.travis-ci.com/user/environment-variables. The report consists of one line per verification: +- Build timestamp - Timestamp of the sketch verification in UTC. - Build - The Travis CI build number. -- Job - Travis CI job number +- Job - Travis CI job number. - Job URL - The URL of the Travis CI job log. -- Build Trigger - The cause of this Travis CI build. Values are `push`, `pull_request`, `api`, `cron`. -- Allow Job Failure - Whether the Travis CI configuration was set to allow the failure of this job without failing the build. +- Build Trigger - The cause of this Travis CI build. Possible values are `push`, `pull_request`, `api`, `cron`. +- Allow Job Failure - Whether the Travis CI configuration was set to allow the failure of this job without failing the build. See: https://docs.travis-ci.com/user/customizing-the-build/#Rows-that-are-Allowed-to-Fail. - PR# - Pull request number (if build was triggered by a pull request). - Branch - The branch of the repository that was built. - Commit - Commit hash of the build. - Commit range - The range of commits that were included in the push or pull request. -- Commit Message - First line of the commit message +- Commit Message - First line of the commit message. - Sketch filename - Board ID - IDE version @@ -133,7 +186,7 @@ Echo a tab separated report of all verification results to the log. The report i - Exit Status - Exit status returned by arduino after the sketch verification. - \# Board Issues - The number of board issues detected. - Board Issue - Short description of the last board issue detected. -- \# Library Issues - The number of library issues detected. Library issues are things that cause warnings in the sketch verification output from the IDE, rather than the compiler. +- \# Library Issues - The number of library issues detected. Library issues are things that cause warnings in the sketch verification output that come from the IDE, rather than the compiler. - Library Issue - Short description of the last library issue detected. ##### `publish_report_to_repository REPORT_GITHUB_TOKEN repositoryURL reportBranch reportFolder doLinkComment` @@ -141,24 +194,25 @@ Add the report to a repository. See the [instructions for publishing job reports - Parameter: **REPORT_GITHUB_TOKEN** - The hidden or encrypted environment variable containing the GitHub personal access token. - Parameter: **repositoryURL** - The .git URL of the repository to publish the report to. This URL can be found by clicking the "Clone or download" button on the home page of the repository. The repository must already exist. - Parameter: **reportBranch** - The branch to publish the report to. The branch must already exist. -- Parameter: **reportFolder** - The the folder to publish the report to. The folder will be created if it doesn't exist. -- Parameter: **doLinkComment** - `true` or `false` Whether to comment on the commit with a link to the report. +- Parameter: **reportFolder** - The folder to publish the report to. The folder will be created if it doesn't exist. +- Parameter: **doLinkComment** - `true` or `false` Whether to comment on the GitHub thread of the commit that triggered the build with a link to the report. ##### `publish_report_to_gist REPORT_GITHUB_TOKEN REPORT_GIST_URL doLinkComment` Add the report to the report gist. See the [instructions for publishing job reports](publishing-job-reports) for details. - Parameter: **REPORT_GITHUB_TOKEN** - The hidden or encrypted environment variable containing the GitHub personal access token. - Parameter: **REPORT_GIST_URL** - The URL of the report gist. -- Parameter: **doLinkComment** - `true` or `false` Whether to comment on the commit with a link to the report. +- Parameter: **doLinkComment** - `true` or `false` Whether to comment on the GitHub thread of the commit that triggered the build with a link to the report. -#### Publishing job reports -The script offers the option of publishing the job result reports to a repository or GitHub [gist](https://gist.github.com/) by using the `publish_report_to_repository` or `publish_report_to_gist` functions. This makes it easier to view the reports or to import them into a spreadsheet program. You also have the option of having the link to the reports automatically added in a comment to the commit being tested. This requires some configuration, which is described in the instructions below. +### Publishing job reports +The script offers the option of publishing the job result reports to a repository or GitHub [gist](https://gist.github.com/) by using the `publish_report_to_repository` or `publish_report_to_gist` functions. This makes it easier to view the reports or to import them into a spreadsheet program. You also have the option of having the link to the reports automatically added in a comment to the commit that triggered the build. This requires some configuration, which is described in the instructions below. -NOTE: For security reasons reports for builds of pull requests from a fork of the repository can not be published. If the owner of that fork wants to publish reports they can create a GitHub token (and gist if using `publish_report_to_gist`) and configure the Travis CI settings for their fork of the repository following these instructions. -##### Creating a GitHub personal access token +NOTE: For security reasons, reports for builds of pull requests from a fork of the repository can not be published. If the owner of that fork wants to publish reports they can create a GitHub token (and gist if using `publish_report_to_gist`) and configure the Travis CI settings for their fork of the repository following these instructions. + +#### Creating a GitHub personal access token This is required for either publishing option. 1. Sign in to your GitHub account. -2. Click your avatar at the top right corner of GitHub > **Settings** > **Personal access tokens** > **Generate new token**. +2. Click your avatar at the top right corner of GitHub > **Settings** > **Developer settings** > **Personal access tokens** > **Generate new token**. 3. Check the appropriate permissions for the token: 1. If using `publish_report_to_gist` check **gist**. 2. If using `publish_report_to_repository` or setting the `doLinkComment` argument of `publish_report_to_gist` check **public_repo** (for public repositories only) or **repo** (for private and public repositories). @@ -169,11 +223,12 @@ This is required for either publishing option. 9. Click the **Add** button. An alternative to using a Travis CI hidden environment variable as described above is to define the GitHub personal access token as an encrypted environment variable: https://docs.travis-ci.com/user/environment-variables/#Encrypting-environment-variables. -##### Creating a gist + +#### Creating a gist This is required for use of the `publish_report_to_gist` function. 1. Open https://gist.github.com/ 2. Sign in to your GitHub account. -3. Type an appropriate name in the **Filename including extension...** field. Gists sort files alphabetically so the filename should be something that will sort before the report filenames, which start at travis_ci_job_report_1.1.tsv. +3. Type an appropriate name in the **Filename including extension...** field. Gists sort files alphabetically so the filename should be something that will sort before the report filenames, which start at travis_ci_job_report_00001.001.tsv. 4. Add some text to the file contents box. 5. Click **Create secret gist** if you don't want the gist to be discoverable (it can still be read by anyone who knows the URL), or **Create public gist** to make it discoverable. 6. Copy the URL of the gist. @@ -182,32 +237,30 @@ This is required for use of the `publish_report_to_gist` function. 9. Click the **Add** button. -#### Troubleshooting +### Troubleshooting ##### Script hangs after an arduino command The Arduino IDE will usually try to start the GUI whenever there is an error in the command. Since the Travis CI build environment does not support this it will just hang for ten minutes until Travis CI automatically cancels the job. This means you get no useful information on the cause of the problem. + ##### Verbose output -Verbose output results in a harder to read log so you should leave it off or minimized when possible. Note that turning on verbose output for a large build may cause the log to exceed 4 MB, which causes Travis CI to terminate the job. -- Verbose script output - Add or uncomment the following lines in your `.travis.yml` file to get more information for troubleshooting. - - Print shell input lines as they are read: - - `- set_verbose_script_output "true"` - - Print a trace of simple commands, for commands, case commands, select commands, and arithmetic for commands and their arguments or associated word lists after they are expanded and before they are executed. The value of the PS4 variable is expanded and the resultant value is printed before the command and its expanded arguments. - - `- set_more_verbose_script_output "true"` -- Verbose output for Travis CI and script - Add one or both of the following lines to your `.travis.yml` file to get more information for troubleshooting of both the Travis CI build process and the script. Do not turn on verbosity by passing `true` to `set_verbose_script_output` or `set_more_verbose_script_output` when you have these lines in your `.travis.yml` file. +Verbose output results in a harder to read log so you should leave it off or minimized when possible but it can be useful for troubleshooting. Note that turning on verbose output for a large build may cause the log to exceed 4 MB, which causes Travis CI to terminate the job. +- Verbose script output - See [`set_script_verbosity` documentation](set_script_verbosity-script_verbosity_level) in the Usage section. +- Verbose output during compilation - See [`set_verbose_output_during_compilation` documentation](set_verbose_output_during_compilation-verboseoutputduringcompilation) in the Usage section. +- Verbose output for Travis CI - Add one or both of the following lines to your `.travis.yml` file to get more details of the Travis CI build process. - Print shell input lines as they are read: - `- set -o verbose` - Print a trace of simple commands, for commands, case commands, select commands, and arithmetic for commands and their arguments or associated word lists after they are expanded and before they are executed. The value of the PS4 variable is expanded and the resultant value is printed before the command and its expanded arguments. - `- set -o xtrace` -- Verbose output during compilation - Add the following line to your `.travis.yml` file to get verbose output from arduino of the commands used in the sketch building process: - - `set_verbose_output_during_compilation true` + ##### Problematic IDE versions Some older versions of the Arduino IDE have bugs or limitations that may cause problems if used with this script: - 1.5.1 and older - The command line interface was added in 1.5.2, thus no version older than that can be used. - 1.5.4 and older - Do not support board-specific parameters, set by custom **Tools** menu items. -- 1.5.5 and older - Do not support setting preferences (`--pref`), thus the sketchbook folder argument of `set_parameters` will not be used. +- 1.5.5 and older - Do not support setting preferences (`--pref`), thus `set_sketchbook_folder` can not be used if no newer IDE version has been installed. - 1.5.5-r2 and older - Don't recognize libraries that have a library.properties` file that doesn't define a `core-dependencies` property. The file include is successful but compilation of sketches that use the library functions will fail. - 1.5.6 and older - `-` or `.` are not allowed in sketch or library folder names. If any are present the Arduino IDE will hang indefinitely when it's executed. -- 1.6.2 - Moves its hardware packages to the .arduino15 folder, causing all other IDE versions to use those cores, some of which are not compatible. For this reason 1.6.2 has been removed from the default list of versions but may still be specified via the `IDE_VERSIONS` argument. -- 1.6.3 and older - Do not support installing boards (`--install-boards`), thus `install_package` can't be used. +- 1.6.2 - Moves its hardware packages to the .arduino15 folder, causing all other IDE versions to use those cores, some of which are not compatible. For this reason 1.6.2 is not installed when a version range containing, but not starting or ending in, 1.6.2 is passed to `install_ide`. 1.6.2 is installed if it is explicitly specified in a version list. +- 1.6.3 and older - Do not support installing boards (`--install-boards`), thus `install_package` can't be used if no newer IDE version has been installed. + -#### Contributing -Pull requests or issue reports are welcome! Please see the [contribution rules](https://github.com/per1234/arduino-ci-script/blob/master/CONTRIBUTING.md) for instructions. +### Contributing +Pull requests or issue reports are welcome! Please see the [contribution rules](https://github.com/per1234/arduino-ci-script/blob/master/.github/CONTRIBUTING.md) for instructions. diff --git a/avr/travis-ci/arduino-ci-script/arduino-ci-script.sh b/avr/travis-ci/arduino-ci-script/arduino-ci-script.sh old mode 100644 new mode 100755 index f83adc01..d96e545b --- a/avr/travis-ci/arduino-ci-script/arduino-ci-script.sh +++ b/avr/travis-ci/arduino-ci-script/arduino-ci-script.sh @@ -2,7 +2,6 @@ # This script is used to automate continuous integration tasks for Arduino projects # https://github.com/per1234/arduino-ci-script - # Based on https://github.com/adafruit/travis-ci-arduino/blob/eeaeaf8fa253465d18785c2bb589e14ea9893f9f/install.sh#L11 # It seems that arrays can't been seen in other functions. So instead I'm setting $IDE_VERSIONS to a string that is the command to create the array readonly ARDUINO_CI_SCRIPT_IDE_VERSION_LIST_ARRAY_DECLARATION="declare -a -r IDEversionListArray=" @@ -22,7 +21,7 @@ readonly ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS=0 readonly ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS=1 # Arduino IDE 1.8.2 and newer generates a ton of garbage output (appears to be something related to jmdns) that must be filtered for the log to be readable and to avoid exceeding the maximum log length -readonly ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX='(^\[SocketListener\(travis-job-*|^ *[0-9][0-9]*: [0-9a-g][0-9a-g]*|^dns\[query,[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*:[0-9][0-9]*, length=[0-9][0-9]*, id=|^dns\[response,[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*:[0-9][0-9]*, length=[0-9][0-9]*, id=|^questions:$|\[DNSQuestion@|^\.\]$|^\.\]\]$|^.\.\]$|^.\.\]\]$)' +readonly ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX='(^\[SocketListener\(travis-job-*|^ *[0-9][0-9]*: [0-9a-g][0-9a-g]*|^dns\[query,[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*:[0-9][0-9]*, length=[0-9][0-9]*, id=|^dns\[response,[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*:[0-9][0-9]*, length=[0-9][0-9]*, id=|^questions:$|\[DNSQuestion@|type: TYPE_IGNORE|^\.\]$|^\.\]\]$|^.\.\]$|^.\.\]\]$)' # Default value ARDUINO_CI_SCRIPT_TOTAL_SKETCH_BUILD_FAILURE_COUNT=0 @@ -34,10 +33,8 @@ else ARDUINO_CI_SCRIPT_ARDUINO_COMMAND="arduino" fi - # Create the folder if it doesn't exist -function create_folder() -{ +function create_folder() { local -r folderName="$1" if ! [[ -d "$folderName" ]]; then # shellcheck disable=SC2086 @@ -45,11 +42,7 @@ function create_folder() fi } - -function set_script_verbosity() -{ - enable_verbosity - +function set_script_verbosity() { ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL="$1" if [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" == "true" ]]; then @@ -75,27 +68,20 @@ function set_script_verbosity() ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT="&>/dev/null" fi - disable_verbosity } - # Deprecated, use set_script_verbosity -function set_verbose_script_output() -{ +function set_verbose_script_output() { set_script_verbosity 1 } - # Deprecated, use set_script_verbosity -function set_more_verbose_script_output() -{ +function set_more_verbose_script_output() { set_script_verbosity 2 } - # Turn on verbosity based on the preferences set by set_script_verbosity -function enable_verbosity() -{ +function enable_verbosity() { # Store previous verbosity settings so they can be set back to their original values at the end of the function shopt -q -o verbose ARDUINO_CI_SCRIPT_PREVIOUS_VERBOSE_SETTING="$?" @@ -115,10 +101,8 @@ function enable_verbosity() fi } - # Return verbosity settings to their previous values -function disable_verbosity() -{ +function disable_verbosity() { if [[ "$ARDUINO_CI_SCRIPT_PREVIOUS_VERBOSE_SETTING" == "0" ]]; then set -o verbose else @@ -132,16 +116,14 @@ function disable_verbosity() fi } - # Verbosity and, in some cases, errexit must be disabled before an early return from a public function, this allows it to be done in a single line instead of two -function return_handler() -{ +function return_handler() { local -r exitStatus="$1" # If exit status is success and errexit is enabled then it must be disabled before exiting the script because errexit must be disabled by default and only enabled in the functions that specifically require it. # If exit status is not success then errexit should not be disabled, otherwise Travis CI won't fail the build even though the exit status was failure. if [[ "$exitStatus" == "$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS" ]] && shopt -q -o errexit; then - set +o errexit + set +o errexit fi disable_verbosity @@ -149,9 +131,7 @@ function return_handler() return "$exitStatus" } - -function set_application_folder() -{ +function set_application_folder() { enable_verbosity ARDUINO_CI_SCRIPT_APPLICATION_FOLDER="$1" @@ -159,9 +139,7 @@ function set_application_folder() disable_verbosity } - -function set_sketchbook_folder() -{ +function set_sketchbook_folder() { enable_verbosity ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER="$1" @@ -177,18 +155,14 @@ function set_sketchbook_folder() disable_verbosity } - # Deprecated -function set_parameters() -{ +function set_parameters() { set_application_folder "$1" set_sketchbook_folder "$2" } - # Check for errors with the board definition that don't affect sketch verification -function set_board_testing() -{ +function set_board_testing() { enable_verbosity ARDUINO_CI_SCRIPT_TEST_BOARD="$1" @@ -196,10 +170,8 @@ function set_board_testing() disable_verbosity } - # Check for errors with libraries that don't affect sketch verification -function set_library_testing() -{ +function set_library_testing() { enable_verbosity ARDUINO_CI_SCRIPT_TEST_LIBRARY="$1" @@ -207,10 +179,8 @@ function set_library_testing() disable_verbosity } - # Install all specified versions of the Arduino IDE -function install_ide() -{ +function install_ide() { enable_verbosity local -r startIDEversion="$1" @@ -220,23 +190,46 @@ function install_ide() # set -o errexit will cause the script to exit as soon as any command returns a non-zero exit status. Without this the success of the function call is determined by the exit status of the last command in the function set -o errexit - # Generate an array declaration string containing a list all Arduino IDE versions which support CLI (1.5.2+ according to https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#history) + if [[ "$ARDUINO_CI_SCRIPT_APPLICATION_FOLDER" == "" ]]; then + echo "ERROR: Application folder was not set. Please use the set_application_folder function to define the location of the application folder." + return_handler "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi + + # Generate an array declaration string containing a list all available Arduino IDE versions which support CLI + # Save the current folder + local -r previousFolder="$PWD" cd "$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" # Create empty local repo for the purpose of getting a list of tags in the arduino/Arduino repository git init --quiet Arduino cd Arduino git remote add origin https://github.com/arduino/Arduino.git if [[ "$startIDEversion" != "1.6.2" ]] && [[ "$startIDEversion" != "1.6.2" ]]; then - # Arduino IDE 1.6.2 has the nasty behavior of moving the included hardware cores to the .arduino15 folder, causing those versions to be used for all builds after Arduino IDE 1.6.2 is used. For that reason, 1.6.2 will only be installed if explicitely specified in the install_ide version arguments + # See "Arduino IDE version blocklist" documentation below local -r IDEversion162regex=--regex='refs/tags/1\.6\.2' if [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" -gt 0 ]]; then echo "NOTE: Due to not playing nicely with other versions, Arduino IDE 1.6.2 will not be installed unless explicitly specified in the version arguments." fi fi - local -r ARDUINO_CI_SCRIPT_FULL_IDE_VERSION_LIST_ARRAY="${ARDUINO_CI_SCRIPT_IDE_VERSION_LIST_ARRAY_DECLARATION}'(\"$(git ls-remote --quiet --tags --refs | grep --invert-match --regexp='refs/tags/1\.0' --regexp='refs/tags/1\.5$' --regexp='refs/tags/1\.5\.1$' --regexp='refs/tags/1\.5\.4-r2$' --regexp='refs/tags/1\.5\.5-r2$' --regexp='refs/tags/1\.5\.7-macosx-java7$' --regexp='refs/tags/1\.5\.8-macosx-java7$' ${IDEversion162regex} --regexp='refs/tags/1\.6\.5-r2$' --regexp='refs/tags/1\.6\.5-r3$' | grep --regexp='refs/tags/[0-9]\+\.[0-9]\+\.[0-9]\+\(\(-.*$\)\|$\)' | cut --delimiter='/' --fields=3 | sort --version-sort | sed ':a;N;$!ba;s/\n/\" \"/g')\")'" + + # Arduino IDE tag blocklist: + # <1.5.2: no CLI (https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#history) + # 1.5.4-r2: Not available for download + # 1.5.5-r2: Not available for download + # 1.5.7-macosx-java7: Not available for download + # 1.5.8-macosx-java7: Not available for download + # 1.6.2: has the nasty behavior of moving the included hardware cores to the .arduino15 folder, causing those versions to be used for all builds after Arduino IDE 1.6.2 is used. For that reason, 1.6.2 will only be installed if explicitly specified in the install_ide version arguments + # 1.6.5-r2: Not available for download + # 1.6.5-r3: Not available for download + # 1.6.5-r2: Not available for download + # 1.6.5-r3: Not available for download + # 1.8.11-ms-store-1: Not available for download + # 1.8.13-ms-store-1: Not available for download + local -r ARDUINO_CI_SCRIPT_FULL_IDE_VERSION_LIST_ARRAY="${ARDUINO_CI_SCRIPT_IDE_VERSION_LIST_ARRAY_DECLARATION}'(\"$(git ls-remote --quiet --tags --refs | grep --invert-match --regexp='refs/tags/1\.0' --regexp='refs/tags/1\.5$' --regexp='refs/tags/1\.5\.1$' --regexp='refs/tags/1\.5\.4-r2$' --regexp='refs/tags/1\.5\.5-r2$' --regexp='refs/tags/1\.5\.7-macosx-java7$' --regexp='refs/tags/1\.5\.8-macosx-java7$' ${IDEversion162regex} --regexp='refs/tags/1\.6\.5-r2$' --regexp='refs/tags/1\.6\.5-r3$' --regexp='refs/tags/.*-ms-store.*$' | grep --regexp='refs/tags/[0-9]\+\.[0-9]\+\.[0-9]\+\(\(-.*$\)\|$\)' | cut --delimiter='/' --fields=3 | sort --version-sort | sed ':a;N;$!ba;s/\n/\" \"/g')\")'" cd .. # Remove the temporary repo rm Arduino --recursive --force + # Go back to the previous folder location + cd "$previousFolder" # Determine list of IDE versions to install generate_ide_version_list_array "$ARDUINO_CI_SCRIPT_FULL_IDE_VERSION_LIST_ARRAY" "$startIDEversion" "$endIDEversion" @@ -246,16 +239,29 @@ function install_ide() determine_ide_version_extremes "$INSTALLED_IDE_VERSION_LIST_ARRAY" NEWEST_INSTALLED_IDE_VERSION="$ARDUINO_CI_SCRIPT_DETERMINED_NEWEST_IDE_VERSION" - if [[ "$ARDUINO_CI_SCRIPT_APPLICATION_FOLDER" == "" ]]; then - echo "ERROR: Application folder was not set. Please use the set_application_folder function to define the location of the application folder." - return_handler "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" - fi create_folder "$ARDUINO_CI_SCRIPT_APPLICATION_FOLDER" # This runs the command contained in the $INSTALLED_IDE_VERSION_LIST_ARRAY string, thus declaring the array locally as $IDEversionListArray. This must be done in any function that uses the array # Dummy declaration to fix the "referenced but not assigned" warning. local IDEversionListArray eval "$INSTALLED_IDE_VERSION_LIST_ARRAY" + + # Determine whether any of the IDE versions to be installed require the creation of a virtual framebuffer (https://github.com/arduino/Arduino/blob/54264124b72eec40aaa22e327c16760f5e806c2a/build/shared/manpage.adoc#bugs) + # This is necessary in Arduino IDE 1.6.13 and older (https://github.com/arduino/Arduino/pull/5578) when running on a headless system + if [ -e /usr/bin/Xvfb ]; then + local -r virtualFramebufferRequiredRegex='^1\.[56]\.' + local IDEversion + for IDEversion in "${IDEversionListArray[@]}"; do + if [[ "$IDEversion" =~ $virtualFramebufferRequiredRegex ]]; then + # based on https://learn.adafruit.com/continuous-integration-arduino-and-you/testing-your-project + /sbin/start-stop-daemon --start $ARDUINO_CI_SCRIPT_QUIET_OPTION $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION --pidfile /tmp/custom_xvfb_1.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :1 -ac -screen 0 1280x1024x16 + sleep 3 + export DISPLAY=:1.0 + break + fi + done + fi + local IDEversion for IDEversion in "${IDEversionListArray[@]}"; do local IDEinstallFolder="$ARDUINO_CI_SCRIPT_APPLICATION_FOLDER/arduino-${IDEversion}" @@ -268,7 +274,7 @@ function install_ide() fi # Determine download file extension - local tgzExtensionVersionsRegex="1.5.[0-9]" + local tgzExtensionVersionsRegex="^1\.5\.[0-9]$" if [[ "$IDEversion" =~ $tgzExtensionVersionsRegex ]]; then # The download file extension prior to 1.6.0 is .tgz local downloadFileExtension="tgz" @@ -283,10 +289,10 @@ function install_ide() local downloadVersion="$IDEversion" fi - wget --no-verbose $ARDUINO_CI_SCRIPT_QUIET_OPTION "http://downloads.arduino.cc/arduino-${downloadVersion}-linux64.${downloadFileExtension}" - tar --extract --file="arduino-${downloadVersion}-linux64.${downloadFileExtension}" - rm $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "arduino-${downloadVersion}-linux64.${downloadFileExtension}" - mv $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "arduino-${downloadVersion}" "$IDEinstallFolder" + wget --no-verbose $ARDUINO_CI_SCRIPT_QUIET_OPTION --directory-prefix="${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/" "http://downloads.arduino.cc/arduino-${downloadVersion}-linux64.${downloadFileExtension}" + tar --extract --directory="$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" --file="${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/arduino-${downloadVersion}-linux64.${downloadFileExtension}" + rm $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/arduino-${downloadVersion}-linux64.${downloadFileExtension}" + mv $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/arduino-${downloadVersion}" "$IDEinstallFolder" fi done @@ -303,11 +309,9 @@ function install_ide() disable_verbosity } - # Generate an array of Arduino IDE versions as a subset of the list provided in the base array defined by the start and end versions # This function allows the same code to be shared by install_ide and build_sketch. The generated array is "returned" as a global named "$ARDUINO_CI_SCRIPT_GENERATED_IDE_VERSION_LIST_ARRAY" -function generate_ide_version_list_array() -{ +function generate_ide_version_list_array() { local -r baseIDEversionArray="$1" local startIDEversion="$2" local endIDEversion="$3" @@ -326,7 +330,6 @@ function generate_ide_version_list_array() endIDEversion="$ARDUINO_CI_SCRIPT_DETERMINED_NEWEST_IDE_VERSION" fi - if [[ "$startIDEversion" == "" || "$startIDEversion" == "all" ]]; then # Use the full base array ARDUINO_CI_SCRIPT_GENERATED_IDE_VERSION_LIST_ARRAY="$baseIDEversionArray" @@ -393,11 +396,9 @@ function generate_ide_version_list_array() fi } - # Determine the oldest and newest (non-hourly unless hourly is the only version on the list) IDE version in the provided array # The determined versions are "returned" by setting the global variables "$ARDUINO_CI_SCRIPT_DETERMINED_OLDEST_IDE_VERSION" and "$ARDUINO_CI_SCRIPT_DETERMINED_NEWEST_IDE_VERSION" -function determine_ide_version_extremes() -{ +function determine_ide_version_extremes() { local -r baseIDEversionArray="$1" # Reset the variables from any value they were assigned the last time the function was ran @@ -417,18 +418,16 @@ function determine_ide_version_extremes() done } - -function set_ide_preference() -{ +function set_ide_preference() { local -r preferenceString="$1" # --pref option is only supported by Arduino IDE 1.5.6 and newer - local -r unsupportedPrefOptionVersionsRegex="1.5.[0-5]" + local -r unsupportedPrefOptionVersionsRegex="^1\.5\.[0-5]$" if ! [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedPrefOptionVersionsRegex ]]; then install_ide_version "$NEWEST_INSTALLED_IDE_VERSION" # --save-prefs was added in Arduino IDE 1.5.8 - local -r unsupportedSavePrefsOptionVersionsRegex="1.5.[6-7]" + local -r unsupportedSavePrefsOptionVersionsRegex="^1\.5\.[6-7]$" if ! [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedSavePrefsOptionVersionsRegex ]]; then # shellcheck disable=SC2086 eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --pref "$preferenceString" --save-prefs "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" @@ -440,19 +439,28 @@ function set_ide_preference() fi } - -function install_ide_version() -{ +function install_ide_version() { local -r IDEversion="$1" - # Create a symbolic link so that the Arduino IDE can always be referenced from the same path no matter which version is being used. - ln --symbolic --force --no-dereference $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/arduino-${IDEversion}" "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}" -} + # Create a symbolic link so that the Arduino IDE can always be referenced by the user from the same path no matter which version is being used. + if [[ "$OSTYPE" == "cygwin" ]] || [[ "$OSTYPE" == "msys" ]]; then + # git-bash's ln just does a copy instead of making a symlink, which takes forever and fails when the target folder exists (despite --force), which takes forever. + # Therefore, use the native Windows command mklink to create a directory junction instead. + # Using a directory junction instead of symlink because supposedly a symlink requires admin privileges. + # Windows doesn't seem to provide any way to overwrite directory junctions + if [[ -d "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}" ]]; then + rm --recursive --force "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER:?}" + fi + # https://stackoverflow.com/a/25394801 + cmd <<<"mklink /J \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}\\${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}\" \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER//\//\\}\\arduino-${IDEversion}\"" >/dev/null + else + ln --symbolic --force --no-dereference $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/arduino-${IDEversion}" "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}" + fi +} # Install hardware packages -function install_package() -{ +function install_package() { enable_verbosity set -o errexit @@ -471,6 +479,7 @@ function install_package() # Clone the repository local -r branchName="$2" + local -r previousFolder="$PWD" cd "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/hardware" if [[ "$branchName" == "" ]]; then @@ -478,7 +487,9 @@ function install_package() else git clone --quiet --branch "$branchName" "$packageURL" fi + cd "$previousFolder" else + local -r previousFolder="$PWD" cd "$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" # Delete everything from the temporary folder @@ -491,10 +502,11 @@ function install_package() extract ./*.* # Delete all files from the temporary folder - find ./ -type f -maxdepth 1 -delete + find ./ -maxdepth 1 -type f -delete # Install the package mv $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION ./* "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/hardware/" + cd "$previousFolder" fi elif [[ "$1" == "" ]]; then @@ -503,10 +515,12 @@ function install_package() local packageName packageName="$(echo "$TRAVIS_REPO_SLUG" | cut -d'/' -f 2)" mkdir --parents $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/hardware/$packageName" + local -r previousFolder="$PWD" cd "$TRAVIS_BUILD_DIR" cp --recursive $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION ./* "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/hardware/${packageName}" # * doesn't copy .travis.yml but that file will be present in the user's installation so it should be there for the tests too cp $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${TRAVIS_BUILD_DIR}/.travis.yml" "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/hardware/${packageName}" + cd "$previousFolder" else # Install package via Boards Manager @@ -521,9 +535,9 @@ function install_package() fi # Check if the newest installed IDE version supports --install-boards - local -r unsupportedInstallBoardsOptionVersionsRange1regex="1.5.[0-9]" - local -r unsupportedInstallBoardsOptionVersionsRange2regex="1.6.[0-3]" - if [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallBoardsOptionVersionsRange1regex || "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallBoardsOptionVersionsRange2regex ]]; then + local -r unsupportedInstallBoardsOptionVersionsRange1regex="^1\.5\.[0-9]$" + local -r unsupportedInstallBoardsOptionVersionsRange2regex="^1\.6\.[0-3]$" + if [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallBoardsOptionVersionsRange1regex ]] || [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallBoardsOptionVersionsRange2regex ]]; then echo "ERROR: --install-boards option is not supported by the newest version of the Arduino IDE you have installed. You must have Arduino IDE 1.6.4 or newer installed to use this function." return_handler "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" else @@ -532,11 +546,51 @@ function install_package() # If defined add the boards manager URL to preferences if [[ "$packageURL" != "" ]]; then + # Get the current Additional Boards Manager URLs preference value so it won't be overwritten when the new URL is added + local priorBoardsmanagerAdditionalURLs + local getPrefExitStatus + # arduino --get-pref returns 4 when the preference does not exist, which is an acceptable circumstance. So it's necessary to unset errexit + set +o errexit + if [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" -eq 0 ]]; then + priorBoardsmanagerAdditionalURLs=$( + "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}" --get-pref boardsmanager.additional.urls 2>/dev/null | tail --lines=1 + exit "${PIPESTATUS[0]}" + ) + getPrefExitStatus="$?" + elif [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" -eq 1 ]]; then + priorBoardsmanagerAdditionalURLs=$( + "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}" --get-pref boardsmanager.additional.urls | tail --lines=1 + exit "${PIPESTATUS[0]}" + ) + getPrefExitStatus="$?" + else + priorBoardsmanagerAdditionalURLs=$( + "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}" --get-pref boardsmanager.additional.urls | tee /dev/tty | tail --lines=1 + exit "${PIPESTATUS[0]}" + ) + getPrefExitStatus="$?" + fi + set -o errexit + + if [[ "$getPrefExitStatus" == "4" ]]; then + # No boardsmanager.additional.urls preference was set. This causes priorBoardsmanagerAdditionalURLs to have a garbage value with Arduino IDE 1.8.10 and newer. + priorBoardsmanagerAdditionalURLs="" + fi + + local -r blankregex="^[ ]*$" + if [[ "$priorBoardsmanagerAdditionalURLs" =~ $blankregex ]]; then + # There is no previous Additional Boards Manager URLs preference value + local boardsmanagerAdditionalURLs="$packageURL" + else + # There is a previous Additional Boards Manager URLs preference value so append the new one to the end of it + local boardsmanagerAdditionalURLs="${priorBoardsmanagerAdditionalURLs},${packageURL}" + fi # grep returns 1 when a line matches the regular expression so it's necessary to unset errexit set +o errexit # shellcheck disable=SC2086 - eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --pref boardsmanager.additional.urls="$packageURL" --save-prefs "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" | tr -Cd '[:print:]\n\t' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX"; local -r arduinoPreferenceSettingExitStatus="${PIPESTATUS[0]}" + eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --pref boardsmanager.additional.urls="$boardsmanagerAdditionalURLs" --save-prefs "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" | tr --complement --delete '[:print:]\n\t' | tr --squeeze-repeats '\n' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX" + local -r arduinoPreferenceSettingExitStatus="${PIPESTATUS[0]}" set -o errexit # this is required because otherwise the exit status of arduino is ignored if [[ "$arduinoPreferenceSettingExitStatus" != "$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS" ]]; then @@ -548,7 +602,8 @@ function install_package() # grep returns 1 when a line matches the regular expression so it's necessary to unset errexit set +o errexit # shellcheck disable=SC2086 - eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --install-boards "$packageID" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" | tr -Cd '[:print:]\n\t' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX"; local -r arduinoInstallPackageExitStatus="${PIPESTATUS[0]}" + eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --install-boards "$packageID" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" | tr --complement --delete '[:print:]\n\t' | tr --squeeze-repeats '\n' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX" + local -r arduinoInstallPackageExitStatus="${PIPESTATUS[0]}" set -o errexit # this is required because otherwise the exit status of arduino is ignored if [[ "$arduinoInstallPackageExitStatus" != "$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS" ]]; then @@ -563,9 +618,7 @@ function install_package() disable_verbosity } - -function install_library() -{ +function install_library() { enable_verbosity set -o errexit @@ -584,6 +637,7 @@ function install_library() local -r branchName="$2" local -r newFolderName="$3" + local -r previousFolder="$PWD" cd "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/libraries" if [[ "$branchName" == "" && "$newFolderName" == "" ]]; then @@ -595,10 +649,12 @@ function install_library() else git clone --quiet --branch "$branchName" "$libraryIdentifier" "$newFolderName" fi + cd "$previousFolder" else # Assume it's a compressed file local -r newFolderName="$2" # Download the file to the temporary folder + local -r previousFolder="$PWD" cd "$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" # Delete everything from the temporary folder @@ -609,10 +665,11 @@ function install_library() extract ./*.* # Delete all files from the temporary folder - find ./ -type f -maxdepth 1 -delete + find ./ -maxdepth 1 -type f -delete # Install the library mv $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION ./* "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/libraries/${newFolderName}" + cd "$previousFolder" fi elif [[ "$libraryIdentifier" == "" ]]; then @@ -621,10 +678,12 @@ function install_library() local libraryName libraryName="$(echo "$TRAVIS_REPO_SLUG" | cut -d'/' -f 2)" mkdir --parents $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/libraries/$libraryName" + local -r previousFolder="$PWD" cd "$TRAVIS_BUILD_DIR" cp --recursive $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION ./* "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/libraries/${libraryName}" # * doesn't copy .travis.yml but that file will be present in the user's installation so it should be there for the tests too cp $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${TRAVIS_BUILD_DIR}/.travis.yml" "${ARDUINO_CI_SCRIPT_SKETCHBOOK_FOLDER}/libraries/${libraryName}" + cd "$previousFolder" else # Install a library that is part of the Library Manager index @@ -636,9 +695,9 @@ function install_library() fi # Check if the newest installed IDE version supports --install-library - local -r unsupportedInstallLibraryOptionVersionsRange1regex="1.5.[0-9]" - local -r unsupportedInstallLibraryOptionVersionsRange2regex="1.6.[0-3]" - if [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallLibraryOptionVersionsRange1regex || "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallLibraryOptionVersionsRange2regex ]]; then + local -r unsupportedInstallLibraryOptionVersionsRange1regex="^1\.5\.[0-9]$" + local -r unsupportedInstallLibraryOptionVersionsRange2regex="^1\.6\.[0-3]$" + if [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallLibraryOptionVersionsRange1regex ]] || [[ "$NEWEST_INSTALLED_IDE_VERSION" =~ $unsupportedInstallLibraryOptionVersionsRange2regex ]]; then echo "ERROR: --install-library option is not supported by the newest version of the Arduino IDE you have installed. You must have Arduino IDE 1.6.4 or newer installed to use this function." return_handler "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" else @@ -647,7 +706,7 @@ function install_library() # Temporarily install the latest IDE version to use for the library installation install_ide_version "$NEWEST_INSTALLED_IDE_VERSION" - # Install the library + # Install the library # shellcheck disable=SC2086 eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --install-library "$libraryName" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" @@ -659,11 +718,9 @@ function install_library() disable_verbosity } - # Extract common file formats # https://github.com/xvoland/Extract -function extract -{ +function extract() { if [ -z "$1" ]; then # display usage if no parameters given echo "Usage: extract ." @@ -671,43 +728,42 @@ function extract return "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" else local filename - for filename in "$@" - do + for filename in "$@"; do if [ -f "$filename" ]; then case "${filename%,}" in - *.tar.bz2|*.tar.gz|*.tar.xz|*.tbz2|*.tgz|*.txz|*.tar) - tar --extract --file="$filename" + *.tar.bz2 | *.tar.gz | *.tar.xz | *.tbz2 | *.tgz | *.txz | *.tar) + tar --extract --file="$filename" ;; - *.lzma) - unlzma $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" + *.lzma) + unlzma $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" ;; - *.bz2) - bunzip2 $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" + *.bz2) + bunzip2 $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" ;; - *.rar) - eval unrar x -ad ./"$filename" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" + *.rar) + eval unrar x -ad ./"$filename" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" ;; - *.gz) - gunzip ./"$filename" + *.gz) + gunzip ./"$filename" ;; - *.zip) - unzip -qq ./"$filename" + *.zip) + unzip -qq ./"$filename" ;; - *.z) - eval uncompress ./"$filename" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" + *.z) + eval uncompress ./"$filename" "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" ;; - *.7z|*.arj|*.cab|*.chm|*.deb|*.dmg|*.iso|*.lzh|*.msi|*.rpm|*.udf|*.wim|*.xar) - 7z x ./"$filename" + *.7z | *.arj | *.cab | *.chm | *.deb | *.dmg | *.iso | *.lzh | *.msi | *.rpm | *.udf | *.wim | *.xar) + 7z x ./"$filename" ;; - *.xz) - unxz $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" + *.xz) + unxz $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" ;; - *.exe) - cabextract $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" + *.exe) + cabextract $ARDUINO_CI_SCRIPT_QUIET_OPTION ./"$filename" ;; - *) - echo "extract: '$filename' - unknown archive method" - return "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + *) + echo "extract: '$filename' - unknown archive method" + return "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ;; esac else @@ -718,9 +774,7 @@ function extract fi } - -function set_verbose_output_during_compilation() -{ +function set_verbose_output_during_compilation() { enable_verbosity local -r verboseOutputDuringCompilation="$1" @@ -733,10 +787,8 @@ function set_verbose_output_during_compilation() disable_verbosity } - # Verify the sketch -function build_sketch() -{ +function build_sketch() { enable_verbosity local -r sketchPath="$1" @@ -752,77 +804,75 @@ function build_sketch() if [[ "$ARDUINO_CI_SCRIPT_GENERATED_IDE_VERSION_LIST_ARRAY" == "$ARDUINO_CI_SCRIPT_IDE_VERSION_LIST_ARRAY_DECLARATION"'()' ]]; then echo "ERROR: The IDE version(s) specified are not installed" - return_handler "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" - fi - - eval "$ARDUINO_CI_SCRIPT_GENERATED_IDE_VERSION_LIST_ARRAY" - local IDEversion - for IDEversion in "${IDEversionListArray[@]}"; do - # Install the IDE - # This must be done before searching for sketches in case the path specified is in the Arduino IDE installation folder - install_ide_version "$IDEversion" - - # The package_index files installed by some versions of the IDE (1.6.5, 1.6.5) can cause compilation to fail for other versions (1.6.5-r4, 1.6.5-r5). Attempting to install a dummy package ensures that the correct version of those files will be installed before the sketch verification. - # Check if the newest installed IDE version supports --install-boards - local unsupportedInstallBoardsOptionVersionsRange1regex="1.5.[0-9]" - local unsupportedInstallBoardsOptionVersionsRange2regex="1.6.[0-3]" - if ! [[ "$IDEversion" =~ $unsupportedInstallBoardsOptionVersionsRange1regex || "$IDEversion" =~ $unsupportedInstallBoardsOptionVersionsRange2regex ]]; then - # shellcheck disable=SC2086 - eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --install-boards arduino:dummy "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" - if [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" -gt 1 ]]; then - # The warning is printed to stdout - echo "NOTE: The warning above \"Selected board is not available\" is caused intentionally and does not indicate a problem." + buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + else + eval "$ARDUINO_CI_SCRIPT_GENERATED_IDE_VERSION_LIST_ARRAY" + local IDEversion + for IDEversion in "${IDEversionListArray[@]}"; do + # Install the IDE + # This must be done before searching for sketches in case the path specified is in the Arduino IDE installation folder + install_ide_version "$IDEversion" + + # The package_index files installed by some versions of the IDE (1.6.5, 1.6.5) can cause compilation to fail for other versions (1.6.5-r4, 1.6.5-r5). Attempting to install a dummy package ensures that the correct version of those files will be installed before the sketch verification. + # Check if the newest installed IDE version supports --install-boards + local unsupportedInstallBoardsOptionVersionsRange1regex="^1\.5\.[0-9]$" + local unsupportedInstallBoardsOptionVersionsRange2regex="^1\.6\.[0-3]$" + if ! [[ "$IDEversion" =~ $unsupportedInstallBoardsOptionVersionsRange1regex ]] && ! [[ "$IDEversion" =~ $unsupportedInstallBoardsOptionVersionsRange2regex ]]; then + # shellcheck disable=SC2086 + eval \"${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}\" --install-boards arduino:dummy "$ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" + if [[ "$ARDUINO_CI_SCRIPT_VERBOSITY_LEVEL" -gt 1 ]]; then + # The warning is printed to stdout + echo "NOTE: The warning above \"Selected board is not available\" is caused intentionally and does not indicate a problem." + fi fi - fi - if [[ "$sketchPath" =~ \.ino$ || "$sketchPath" =~ \.pde$ ]]; then - # A sketch was specified - if ! [[ -f "$sketchPath" ]]; then - echo "ERROR: Specified sketch: $sketchPath doesn't exist" - buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" - elif ! build_this_sketch "$sketchPath" "$boardID" "$IDEversion" "$allowFail"; then - # build_this_sketch returned a non-zero exit status - buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" - fi - else - # Search for all sketches in the path and put them in an array - local sketchFound="false" - # https://github.com/koalaman/shellcheck/wiki/SC2207 - declare -a sketches - mapfile -t sketches < <(find "$sketchPath" -name "*.pde" -o -name "*.ino") - local sketchName - for sketchName in "${sketches[@]}"; do - # Only verify the sketch that matches the name of the sketch folder, otherwise it will cause redundant verifications for sketches that have multiple .ino files - local sketchFolder - sketchFolder="$(echo "$sketchName" | rev | cut -d'/' -f 2 | rev)" - local sketchNameWithoutPathWithExtension - sketchNameWithoutPathWithExtension="$(echo "$sketchName" | rev | cut -d'/' -f 1 | rev)" - local sketchNameWithoutPathWithoutExtension - sketchNameWithoutPathWithoutExtension="${sketchNameWithoutPathWithExtension%.*}" - if [[ "$sketchFolder" == "$sketchNameWithoutPathWithoutExtension" ]]; then - sketchFound="true" - if ! build_this_sketch "$sketchName" "$boardID" "$IDEversion" "$allowFail"; then - # build_this_sketch returned a non-zero exit status - buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" - fi + if [[ "$sketchPath" =~ \.ino$ ]] || [[ "$sketchPath" =~ \.pde$ ]]; then + # A sketch was specified + if ! [[ -f "$sketchPath" ]]; then + echo "ERROR: Specified sketch: $sketchPath doesn't exist" + buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + elif ! build_this_sketch "$sketchPath" "$boardID" "$IDEversion" "$allowFail"; then + # build_this_sketch returned a non-zero exit status + buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" fi - done + else + # Search for all sketches in the path and put them in an array + local sketchFound="false" + # https://github.com/koalaman/shellcheck/wiki/SC2207 + declare -a sketches + mapfile -t sketches < <(find "$sketchPath" -name "*.pde" -o -name "*.ino") + local sketchName + for sketchName in "${sketches[@]}"; do + # Only verify the sketch that matches the name of the sketch folder, otherwise it will cause redundant verifications for sketches that have multiple .ino files + local sketchFolder + sketchFolder="$(echo "$sketchName" | rev | cut -d'/' -f 2 | rev)" + local sketchNameWithoutPathWithExtension + sketchNameWithoutPathWithExtension="$(echo "$sketchName" | rev | cut -d'/' -f 1 | rev)" + local sketchNameWithoutPathWithoutExtension + sketchNameWithoutPathWithoutExtension="${sketchNameWithoutPathWithExtension%.*}" + if [[ "$sketchFolder" == "$sketchNameWithoutPathWithoutExtension" ]]; then + sketchFound="true" + if ! build_this_sketch "$sketchName" "$boardID" "$IDEversion" "$allowFail"; then + # build_this_sketch returned a non-zero exit status + buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi + fi + done - if [[ "$sketchFound" == "false" ]]; then - echo "ERROR: No valid sketches were found in the specified path" - buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + if [[ "$sketchFound" == "false" ]]; then + echo "ERROR: No valid sketches were found in the specified path" + buildSketchExitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi fi - fi - done + done + fi disable_verbosity return $buildSketchExitStatus } - -function build_this_sketch() -{ +function build_this_sketch() { # Fold this section of output in the Travis CI build log to make it easier to read echo -e "travis_fold:start:build_sketch" @@ -832,12 +882,15 @@ function build_this_sketch() local -r allowFail="$4" # Produce a useful label for the fold in the Travis log for this function call - echo "build_sketch $sketchName $boardID $IDEversion $allowFail" + echo "build_sketch $sketchName $boardID $allowFail $IDEversion" # Arduino IDE 1.8.0 and 1.8.1 fail to verify a sketch if the absolute path to it is not specified # http://stackoverflow.com/a/3915420/7059512 local absoluteSketchName - absoluteSketchName="$(cd "$(dirname "$1")"; pwd)/$(basename "$1")" + absoluteSketchName="$( + cd "$(dirname "$1")" + pwd + )/$(basename "$1")" # Define a dummy value for arduinoExitStatus so that the while loop will run at least once local arduinoExitStatus=255 @@ -845,7 +898,8 @@ function build_this_sketch() while [[ $arduinoExitStatus -gt $ARDUINO_CI_SCRIPT_HIGHEST_ACCEPTABLE_ARDUINO_EXIT_STATUS && $verifyCount -le $ARDUINO_CI_SCRIPT_SKETCH_VERIFY_RETRIES ]]; do # Verify the sketch # shellcheck disable=SC2086 - "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}" $ARDUINO_CI_SCRIPT_DETERMINED_VERBOSE_BUILD --verify "$absoluteSketchName" --board "$boardID" 2>&1 | tr -Cd '[:print:]\n\t' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX" | tee "$ARDUINO_CI_SCRIPT_VERIFICATION_OUTPUT_FILENAME"; local arduinoExitStatus="${PIPESTATUS[0]}" + "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/${ARDUINO_CI_SCRIPT_ARDUINO_COMMAND}" $ARDUINO_CI_SCRIPT_DETERMINED_VERBOSE_BUILD --verify "$absoluteSketchName" --board "$boardID" 2>&1 | tr --complement --delete '[:print:]\n\t' | tr --squeeze-repeats '\n' | grep --extended-regexp --invert-match "$ARDUINO_CI_SCRIPT_ARDUINO_OUTPUT_FILTER_REGEX" | tee "$ARDUINO_CI_SCRIPT_VERIFICATION_OUTPUT_FILENAME" + local arduinoExitStatus="${PIPESTATUS[0]}" local verifyCount=$((verifyCount + 1)) done @@ -863,37 +917,37 @@ function build_this_sketch() while read -r outputFileLine; do # Determine program storage memory usage local programStorageRegex="Sketch uses ([0-9,]+) *" - if [[ "$outputFileLine" =~ $programStorageRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $programStorageRegex ]] >/dev/null; then local -r programStorageWithComma=${BASH_REMATCH[1]} fi # Determine dynamic memory usage local dynamicMemoryRegex="Global variables use ([0-9,]+) *" - if [[ "$outputFileLine" =~ $dynamicMemoryRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $dynamicMemoryRegex ]] >/dev/null; then local -r dynamicMemoryWithComma=${BASH_REMATCH[1]} fi # Increment warning count local warningRegex="warning: " - if [[ "$outputFileLine" =~ $warningRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $warningRegex ]] >/dev/null; then warningCount=$((warningCount + 1)) fi # Check for board issues local bootloaderMissingRegex="Bootloader file specified but missing: " - if [[ "$outputFileLine" =~ $bootloaderMissingRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $bootloaderMissingRegex ]] >/dev/null; then local boardIssue="missing bootloader" boardIssueCount=$((boardIssueCount + 1)) fi local boardsDotTxtMissingRegex="Could not find boards.txt" - if [[ "$outputFileLine" =~ $boardsDotTxtMissingRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $boardsDotTxtMissingRegex ]] >/dev/null; then local boardIssue="Could not find boards.txt" boardIssueCount=$((boardIssueCount + 1)) fi local buildDotBoardNotDefinedRegex="doesn't define a 'build.board' preference" - if [[ "$outputFileLine" =~ $buildDotBoardNotDefinedRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $buildDotBoardNotDefinedRegex ]] >/dev/null; then local boardIssue="doesn't define a 'build.board' preference" boardIssueCount=$((boardIssueCount + 1)) fi @@ -902,78 +956,78 @@ function build_this_sketch() # This is the generic "invalid library" warning that doesn't specify the reason local invalidLibrarRegex1="Invalid library found in" local invalidLibrarRegex2="from library$" - if [[ "$outputFileLine" =~ $invalidLibrarRegex1 ]] && ! [[ "$outputFileLine" =~ $invalidLibrarRegex2 ]] > /dev/null; then + if [[ "$outputFileLine" =~ $invalidLibrarRegex1 ]] && ! [[ "$outputFileLine" =~ $invalidLibrarRegex2 ]] >/dev/null; then local libraryIssue="Invalid library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingNameRegex="Invalid library found in .* Missing 'name' from library" - if [[ "$outputFileLine" =~ $missingNameRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingNameRegex ]] >/dev/null; then local libraryIssue="Missing 'name' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingVersionRegex="Invalid library found in .* Missing 'version' from library" - if [[ "$outputFileLine" =~ $missingVersionRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingVersionRegex ]] >/dev/null; then local libraryIssue="Missing 'version' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingAuthorRegex="Invalid library found in .* Missing 'author' from library" - if [[ "$outputFileLine" =~ $missingAuthorRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingAuthorRegex ]] >/dev/null; then local libraryIssue="Missing 'author' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingMaintainerRegex="Invalid library found in .* Missing 'maintainer' from library" - if [[ "$outputFileLine" =~ $missingMaintainerRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingMaintainerRegex ]] >/dev/null; then local libraryIssue="Missing 'maintainer' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingSentenceRegex="Invalid library found in .* Missing 'sentence' from library" - if [[ "$outputFileLine" =~ $missingSentenceRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingSentenceRegex ]] >/dev/null; then local libraryIssue="Missing 'sentence' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingParagraphRegex="Invalid library found in .* Missing 'paragraph' from library" - if [[ "$outputFileLine" =~ $missingParagraphRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingParagraphRegex ]] >/dev/null; then local libraryIssue="Missing 'paragraph' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local missingURLregex="Invalid library found in .* Missing 'url' from library" - if [[ "$outputFileLine" =~ $missingURLregex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $missingURLregex ]] >/dev/null; then local libraryIssue="Missing 'url' from library" libraryIssueCount=$((libraryIssueCount + 1)) fi local invalidVersionRegex="Invalid version found:" - if [[ "$outputFileLine" =~ $invalidVersionRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $invalidVersionRegex ]] >/dev/null; then local libraryIssue="Invalid version found:" libraryIssueCount=$((libraryIssueCount + 1)) fi local invalidCategoryRegex="is not valid. Setting to 'Uncategorized'" - if [[ "$outputFileLine" =~ $invalidCategoryRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $invalidCategoryRegex ]] >/dev/null; then local libraryIssue="Invalid category" libraryIssueCount=$((libraryIssueCount + 1)) fi local spuriousFolderRegex="WARNING: Spurious" - if [[ "$outputFileLine" =~ $spuriousFolderRegex ]] > /dev/null; then + if [[ "$outputFileLine" =~ $spuriousFolderRegex ]] >/dev/null; then local libraryIssue="Spurious folder" libraryIssueCount=$((libraryIssueCount + 1)) fi - done < "$ARDUINO_CI_SCRIPT_VERIFICATION_OUTPUT_FILENAME" + done <"$ARDUINO_CI_SCRIPT_VERIFICATION_OUTPUT_FILENAME" rm $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "$ARDUINO_CI_SCRIPT_VERIFICATION_OUTPUT_FILENAME" # Remove the stupid comma from the memory values if present - local -r programStorage=${programStorageWithComma//,} - local -r dynamicMemory=${dynamicMemoryWithComma//,} + local -r programStorage=${programStorageWithComma//,/} + local -r dynamicMemory=${dynamicMemoryWithComma//,/} if [[ "$boardIssue" != "" && "$ARDUINO_CI_SCRIPT_TEST_BOARD" == "true" ]]; then # There was a board issue and board testing is enabled so fail the build @@ -987,7 +1041,7 @@ function build_this_sketch() fi # Add the build data to the report file - echo "$(date -u "+%Y-%m-%d %H:%M:%S")"$'\t'"$TRAVIS_BUILD_NUMBER"$'\t'"$TRAVIS_JOB_NUMBER"$'\t'"https://travis-ci.org/${TRAVIS_REPO_SLUG}/jobs/${TRAVIS_JOB_ID}"$'\t'"$TRAVIS_EVENT_TYPE"$'\t'"$TRAVIS_ALLOW_FAILURE"$'\t'"$TRAVIS_PULL_REQUEST"$'\t'"$TRAVIS_BRANCH"$'\t'"$TRAVIS_COMMIT"$'\t'"$TRAVIS_COMMIT_RANGE"$'\t'"${TRAVIS_COMMIT_MESSAGE%%$'\n'*}"$'\t'"$sketchName"$'\t'"$boardID"$'\t'"$IDEversion"$'\t'"$programStorage"$'\t'"$dynamicMemory"$'\t'"$warningCount"$'\t'"$allowFail"$'\t'"$arduinoExitStatus"$'\t'"$boardIssueCount"$'\t'"$boardIssue"$'\t'"$libraryIssueCount"$'\t'"$libraryIssue"$'\r' >> "$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" + echo "$(date -u "+%Y-%m-%d %H:%M:%S")"$'\t'"$TRAVIS_BUILD_NUMBER"$'\t'"$TRAVIS_JOB_NUMBER"$'\t'"https://travis-ci.org/${TRAVIS_REPO_SLUG}/jobs/${TRAVIS_JOB_ID}"$'\t'"$TRAVIS_EVENT_TYPE"$'\t'"$TRAVIS_ALLOW_FAILURE"$'\t'"$TRAVIS_PULL_REQUEST"$'\t'"$TRAVIS_BRANCH"$'\t'"$TRAVIS_COMMIT"$'\t'"$TRAVIS_COMMIT_RANGE"$'\t'"${TRAVIS_COMMIT_MESSAGE%%$'\n'*}"$'\t'"$sketchName"$'\t'"$boardID"$'\t'"$IDEversion"$'\t'"$programStorage"$'\t'"$dynamicMemory"$'\t'"$warningCount"$'\t'"$allowFail"$'\t'"$arduinoExitStatus"$'\t'"$boardIssueCount"$'\t'"$boardIssue"$'\t'"$libraryIssueCount"$'\t'"$libraryIssue"$'\r' >>"$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" # Adjust the exit status according to the allowFail setting if [[ "$buildThisSketchExitStatus" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" && ("$allowFail" == "true" || "$allowFail" == "require") ]]; then @@ -1012,10 +1066,8 @@ function build_this_sketch() return $buildThisSketchExitStatus } - # Print the contents of the report file -function display_report() -{ +function display_report() { enable_verbosity if [ -e "$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" ]; then @@ -1034,10 +1086,8 @@ function display_report() disable_verbosity } - # Add the report file to a Git repository -function publish_report_to_repository() -{ +function publish_report_to_repository() { enable_verbosity local -r token="$1" @@ -1053,6 +1103,7 @@ function publish_report_to_repository() # Clone was successful create_folder "${HOME}/report-repository/${reportFolder}" cp $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" "${HOME}/report-repository/${reportFolder}" + local -r previousFolder="$PWD" cd "${HOME}/report-repository" git add $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "${HOME}/report-repository/${reportFolder}/${ARDUINO_CI_SCRIPT_REPORT_FILENAME}" git config user.email "arduino-ci-script@nospam.me" @@ -1076,6 +1127,7 @@ function publish_report_to_repository() git push $ARDUINO_CI_SCRIPT_QUIET_OPTION $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION "https://${token}@${repositoryURL#*//}" gitPushExitStatus="$?" done + cd "$previousFolder" rm --recursive --force "${HOME}/report-repository" if [[ "$gitPushExitStatus" == "$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS" ]]; then if [[ "$doLinkComment" == "true" ]]; then @@ -1114,10 +1166,8 @@ function publish_report_to_repository() disable_verbosity } - # Add the report file to a gist -function publish_report_to_gist() -{ +function publish_report_to_gist() { enable_verbosity local -r token="$1" @@ -1137,7 +1187,7 @@ function publish_report_to_gist() reportContent=$(sed -e 's/\r//' -e's/\t/\\t/g' -e 's/"/\\"/g' "$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" | awk '{ printf($0 "\\n") }') # Upload the report to the Gist. I have to use the here document to avoid the "Argument list too long" error from curl with long reports. Redirect output to dev/null because it dumps the whole gist to the log - eval curl --header "\"Authorization: token ${token}\"" --data @- "\"https://api.github.com/gists/${gistID}\"" < Examples > INCOMPATIBLE." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_ARCHITECTURES_EMPTY_EXIT_STATUS) + else + # Check for invalid architectures + local validArchitecturesRegex='^((\*)|(ameba)|(arm)|(arc32)|(avr)|(efm32)|(esp32)|(esp8266)|(FP51)|(i586)|(i686)|(iot2000)|(mbed)|(megaavr)|(mraa)|(msp430)|(navspark)|(nRF5)|(nRF51822)|(nrf52)|(nRF52832)|(particle-photon)|(particle-electron)|(particle-core)|(pic)|(pic32)|(RFduino)|(sam)|(samd)|(samd_beta)|(Seeed_STM32F4)|(Simblee)|(solox)|(stm)|(STM32)|(stm32)|(STM32F1)|(STM32F2)|(STM32F3)|(STM32F4)|(stm32f4)|(STM32L1)|(STM32L4)|(teensy)|(win10)|(x86))$' + # Split string on , + IFS=',' + local validArchitectureFound=false + # Disable globbing, otherwise it fails when one of the architecture values is * + set -o noglob + # Check for * architecture. If this is found then the other architecture values don't matter + local wildcardArchitectureFound=false + for rawArchitecture in $architecturesValue; do + # The Arduino IDE ignores leading or trailing whitespace on architectures + # Strip leading whitespace + local architecture="${rawArchitecture#"${rawArchitecture%%[![:space:]]*}"}" + # Strip trailing whitespace + architecture="${architecture%"${architecture##*[![:space:]]}"}" + + if [[ "$architecture" == "*" ]]; then + wildcardArchitectureFound=true + validArchitectureFound=true + break + fi + done + if [[ "$wildcardArchitectureFound" == false ]]; then + for rawArchitecture in $architecturesValue; do + # The Arduino IDE ignores leading or trailing whitespace on architectures + # Strip leading whitespace + local architecture="${rawArchitecture#"${rawArchitecture%%[![:space:]]*}"}" + # Strip trailing whitespace + architecture="${architecture%"${architecture##*[![:space:]]}"}" + + if [[ "$architecture" =~ $validArchitecturesRegex ]]; then + validArchitectureFound=true + else + local aliasCheckPassed=true + # If an architecture alias is used then the correct architecture must also be present + check_architecture_alias "$architecture" '^((Avr)|(AVR)|([aA][tT][mM][eE][lL].?[aA][vV][rR]))$' "$architecturesValue" 'avr' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Sam)|(SAM))$' "$architecturesValue" 'sam' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Samd)|(SAMD)|(SamD)|((samD)|([aA][tT][mM][eE][lL].?[sS][aA][mM]))$' "$architecturesValue" 'samd' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Arc32)|(ARC32)|([aA][rR][cC].32)|([iI][nN][tT][eE][lL].?[aA][rR][cC]32))$' "$architecturesValue" 'arc32' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Esp8266)|(ESP8266)|([eE][sS][pP].8266)|(8266)|([eE][sS][pP])|([eE][sS][pP][rR][eE][sS][sS][iI][fF].?(8266)?))$' "$architecturesValue" 'esp8266' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Esp32)|(ESP32)|([eE][sS][pP].32)|(arduino-esp32)|([eE][sS][pP][rR][eE][sS][sS][iI][fF].?32))$' "$architecturesValue" 'esp32' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + check_architecture_alias "$architecture" '^((Teensy)|(TEENSY))$' "$architecturesValue" 'teensy' "$normalizedLibraryPropertiesPath" + if [[ "$?" == "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" ]]; then + aliasCheckPassed=false + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + + if [[ "$aliasCheckPassed" == true ]]; then + echo "WARNING: ${normalizedLibraryPropertiesPath}/library.properties: architectures field contains an unknown architecture: ${architecture}. Note: architecture values are case-sensitive." + fi + fi + done + fi + # Re-enable globbing + set +o noglob + # Set IFS back to default + unset IFS + + # At least one known architecture must be present + if [[ "$validArchitectureFound" == false ]]; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: architectures field (${architecturesValue}) doesn't contain any known architecture values." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_ARCHITECTURE_EXIT_STATUS) + fi + fi + fi + + # Check for invalid lines (anything other than property, comment, or blank line) + if grep --quiet --invert-match --extended-regexp --regexp='=' --regexp='^[[:space:]]*(#|$)' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Invalid line found. Installation of a library with invalid line will cause all compilations to fail. library.properties must only consist of property definitions, blank lines, and comments (#)." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INVALID_LINE_EXIT_STATUS) + fi + + # Check for incorrect includes field name case + if ! check_field_name_case "$libraryProperties" 'includes' "$normalizedLibraryPropertiesPath"; then + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INCLUDES_MISSPELLED_EXIT_STATUS) + fi + + # Check for misspelled includes field name + if grep --quiet --ignore-case --regexp='^[[:space:]]*include[[:space:]]*=' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Misspelled includes field name. It must be spelled exactly \"includes\". See https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_INCLUDES_MISSPELLED_EXIT_STATUS) + fi + + # Check for empty includes value + if grep --quiet --regexp='^[[:space:]]*includes[[:space:]]*=[[:space:]]*$' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Undefined includes field. Either define the field or remove it. See https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_EMPTY_INCLUDES_EXIT_STATUS) + fi + + # Check for incorrect dot_a_linkage field name case + if ! check_field_name_case "$libraryProperties" 'dot_a_linkage' "$normalizedLibraryPropertiesPath"; then + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_DOT_A_LINKAGE_MISSPELLED_EXIT_STATUS) + fi + + # Check for misspelled dot_a_linkage field name + if grep --quiet --ignore-case --extended-regexp --regexp='^[[:space:]]*((dot_a_linkages)|(dot-?a-?linkages?))[[:space:]]*=' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Misspelled dot_a_linkage field name. It must be spelled exactly \"dot_a_linkage\". See https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_DOT_A_LINKAGE_MISSPELLED_EXIT_STATUS) + fi + + # Check for incorrect precompiled field name case + if ! check_field_name_case "$libraryProperties" 'precompiled' "$normalizedLibraryPropertiesPath"; then + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_PRECOMPILED_MISSPELLED_EXIT_STATUS) + fi + + # Check for misspelled precompiled field name + if grep --quiet --ignore-case --extended-regexp --regexp='^[[:space:]]*((precompile)|(pre[-_]compiled?))[[:space:]]*=' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Misspelled precompiled field name. It must be spelled exactly \"precompiled\". See https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_PRECOMPILED_MISSPELLED_EXIT_STATUS) + fi + + # Check for incorrect ldflags field name case + if ! check_field_name_case "$libraryProperties" 'ldflags' "$normalizedLibraryPropertiesPath"; then + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_LDFLAGS_MISSPELLED_EXIT_STATUS) + fi + + # Check for misspelled ldflags field name + if grep --quiet --ignore-case --extended-regexp --regexp='^[[:space:]]*((ldflag)|(ld[-_]flags?))[[:space:]]*=' <<<"$libraryProperties"; then + echo "ERROR: ${normalizedLibraryPropertiesPath}/library.properties: Misspelled ldflags field name. It must be spelled exactly \"ldflags\". See https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_LDFLAGS_MISSPELLED_EXIT_STATUS) + fi + + done <<<"$(find "$normalizedLibraryPropertiesSearchPath" -maxdepth "$maximumSearchDepth" -type d | sort --dictionary-order)" + + return "$exitStatus" +} + +# https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#keywords +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=1 +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCONSEQUENTIAL_MULTIPLE_TABS_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_LINE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCONSEQUENTIAL_LEADING_SPACE_ON_KEYWORD_TOKENTYPE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_MULTIPLE_TABS_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_LEADING_SPACE_ON_KEYWORD_TOKENTYPE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_FOLDER_DOESNT_EXIST_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_MISSPELLED_FILENAME_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCORRECT_FILENAME_CASE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_FIELD_SEPARATOR_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_BOM_CORRUPTED_KEYWORD_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_KEYWORD_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_KEYWORD_TOKENTYPE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_LEADING_SPACE_ON_RSYNTAXTEXTAREA_TOKENTYPE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_RSYNTAXTEXTAREA_TOKENTYPE_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_REFERENCE_LINK_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +function check_keywords_txt() { + local -r keywordsTxtSearchPath="$1" + local maximumSearchDepth="$2" + if [[ "$maximumSearchDepth" == "" ]]; then + # Set default search depth + maximumSearchDepth=0 + fi + + local exitStatus=$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS + + # Replace backslashes with slashes + local -r keywordsTxtSearchPathWithSlashes="${keywordsTxtSearchPath//\\//}" + local -r keywordsTxtRegex='[kK][eE][yY][wW][oO][rR][dD][sS]\.[tT][xX][tT]$' + if [[ ! -d "$keywordsTxtSearchPathWithSlashes" && "$keywordsTxtSearchPathWithSlashes" =~ $keywordsTxtRegex ]]; then + # Path contains the filename but we only want the folder + local -r keywordsTxtSearchPathWithoutFile="${keywordsTxtSearchPathWithSlashes::-12}" + else + local -r keywordsTxtSearchPathWithoutFile="$keywordsTxtSearchPathWithSlashes" + fi + # Remove trailing slash + local -r normalizedKeywordsTxtSearchPath="${keywordsTxtSearchPathWithoutFile%/}" + + # Check whether folder exists + if [[ ! -d "$normalizedKeywordsTxtSearchPath" ]]; then + echo "ERROR: ${normalizedKeywordsTxtSearchPath}: Folder doesn't exist." + return $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_FOLDER_DOESNT_EXIST_EXIT_STATUS + fi + + while read -r normalizedKeywordsTxtPath; do + # Check for misspelled keywords.txt filename + if [[ "$(find "$normalizedKeywordsTxtPath" -type f -iname 'keyword.txt')" || "$(find "$normalizedKeywordsTxtPath" -type f -iregex '.*/keywords?\.text')" || "$(find "$normalizedKeywordsTxtPath" -type f -iregex '.*/keywords?\.txt\.txt')" ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}: Incorrectly spelled keywords.txt file. It must be spelled exactly \"keywords.txt\"." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_MISSPELLED_FILENAME_EXIT_STATUS) + fi + + # Check for incorrect filename case + local keywordsTxtFound=false + while read -r foundKeywordsTxtPath; do + # The while loop always runs once, even if no file was found + if [[ "$foundKeywordsTxtPath" == "" ]]; then + continue + fi + + if [[ "${foundKeywordsTxtPath: -12}" == 'keywords.txt' ]]; then + keywordsTxtFound=true + else + echo "ERROR: ${foundKeywordsTxtPath}: Incorrect filename case, which causes it to not be recognized on a filename case-sensitive OS such as Linux. It must be exactly \"keywords.txt\"." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCORRECT_FILENAME_CASE_EXIT_STATUS) + fi + done <<<"$(find "$normalizedKeywordsTxtPath" -maxdepth 1 -type f -iname 'keywords.txt' | sort --dictionary-order)" + + # Check whether the folder contains a keywords.txt file + if [[ "$keywordsTxtFound" == false ]]; then + # no point in doing any more checks on this folder + continue + fi + + # Read the keywords.txt file line by line + # Split into lines by CR + while IFS='' read -d $'\r' -r keywordsTxtCRline || [[ -n "$keywordsTxtCRline" ]]; do + # Split into lines by LN + while IFS='' read -r keywordsTxtLine || [[ -n "$keywordsTxtLine" ]]; do + # Skip blank lines and comments + local blankLineRegex='^[[:space:]]*$' + local commentRegex='^[[:space:]]*#' + if [[ "$keywordsTxtLine" =~ $blankLineRegex ]] || [[ "$keywordsTxtLine" =~ $commentRegex ]]; then + continue + fi + + # Skip BOM corrupted blank lines and comments + if grep --quiet $'\xEF\xBB\xBF' <<<"$keywordsTxtLine"; then + local BOMcorruptedCommentRegex='^.[[:space:]]*#' + local BOMcorruptedBlankLineRegex='^.[[:space:]]*$' + if [[ "$keywordsTxtLine" =~ $BOMcorruptedCommentRegex ]] || [[ "$keywordsTxtLine" =~ $BOMcorruptedBlankLineRegex ]]; then + echo "WARNING: ${normalizedKeywordsTxtPath}/keywords.txt: BOM found. In this case it does not cause an issue but it's recommended to use UTF-8 encoding for keywords.txt." + continue + fi + fi + + local spacesSeparatorRegex='^[[:space:]]*[^[:space:]]+ +[^[:space:]]+' + # Check for invalid separator + if [[ "$keywordsTxtLine" =~ $spacesSeparatorRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Space(s) used as a field separator. Fields must be separated by a single true tab." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_FIELD_SEPARATOR_EXIT_STATUS) + # The rest of the checks will be borked by messed up field separators so continue to the next line + continue + fi + + # Check for multiple tabs used as separator where this causes unintended results + local consequentialMultipleSeparatorRegex='^[[:space:]]*[^[:space:]]+[[:space:]]*'$'\t''+[[:space:]]*'$'\t''+[[:space:]]*((KEYWORD1)|(LITERAL1))' + if [[ "$keywordsTxtLine" =~ $consequentialMultipleSeparatorRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Multiple tabs used as field separator. It must be a single tab. This causes the default keyword highlighting (as used by KEYWORD2, KEYWORD3, LITERAL2) to be used rather than the intended highlighting." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_MULTIPLE_TABS_EXIT_STATUS) + # The rest of the checks will be borked by messed up field separators so continue to the next line + continue + fi + + # Check for multiple tabs used as separator where this causes no unintended results + local inconsequentialMultipleSeparatorRegex='^[[:space:]]*[^[:space:]]+[[:space:]]*'$'\t''+[[:space:]]*'$'\t''+[[:space:]]*((KEYWORD2)|(KEYWORD3)|(LITERAL2))' + if [[ "$keywordsTxtLine" =~ $inconsequentialMultipleSeparatorRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Multiple tabs used as field separator. It must be a single tab. This causes the default keyword highlighting (as used by KEYWORD2, KEYWORD3, LITERAL2). In this case that doesn't cause the keywords to be colored other than intended but it's recommended to fully comply with the Arduino library specification." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCONSEQUENTIAL_MULTIPLE_TABS_EXIT_STATUS) + # The rest of the checks will be borked by messed up field separators so continue to the next line + continue + fi + + # Check for invalid line + local invalidLineRegex='^[[:space:]]*[^[:space:]]+[[:space:]]*$' + if [[ "$keywordsTxtLine" =~ $invalidLineRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Invalid line. If this was intended as a comment, it should use the correct # syntax." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_LINE_EXIT_STATUS) + # The rest of the checks are pointless so continue to the next line of keywords.txt + continue + fi + + # Get the field values + # Use a unique, non-whitespace field separator character + fieldSeparator=$'\a' + IFS=$fieldSeparator + # Strip leading whitespace. This is ignored by the Arduino IDE (even tabs) + local keywordsTxtLineFrontStripped="${keywordsTxtLine#"${keywordsTxtLine%%[![:space:]]*}"}" + # Change tabs to the field separator character for line splitting + # shellcheck disable=SC2206 + local keywordsTxtLineSwappedTabs=(${keywordsTxtLineFrontStripped//$'\t'/$fieldSeparator}) + + # KEYWORD is the 1st field + local keywordRaw=${keywordsTxtLineSwappedTabs[0]} + # Strip trailing spaces + local keyword="${keywordRaw%"${keywordRaw##*[! ]}"}" + + # KEYWORD_TOKENTYPE is the 2nd field + local keywordTokentypeRaw=${keywordsTxtLineSwappedTabs[1]} + # The Arduino IDE strips trailing spaces from KEYWORD_TOKENTYPE + # Strip trailing spaces + local keywordTokentype="${keywordTokentypeRaw%"${keywordTokentypeRaw##*[! ]}"}" + + # REFERENCE_LINK is the 3rd field + local referenceLinkRaw=${keywordsTxtLineSwappedTabs[2]} + # The Arduino IDE strips leading and trailing whitespace from REFERENCE_LINK + # Strip leading spaces + local referenceLinkFrontStripped="${referenceLinkRaw#"${referenceLinkRaw%%[! ]*}"}" + # Strip trailing spaces + local referenceLink="${referenceLinkFrontStripped%"${referenceLinkFrontStripped##*[! ]}"}" + + # RSYNTAXTEXTAREA_TOKENTYPE is the 4th field + local rsyntaxtextareaTokentypeRaw=${keywordsTxtLineSwappedTabs[3]} + # The Arduino IDE strips trailing spaces from RSYNTAXTEXTAREA_TOKENTYPE + # Strip trailing spaces + local rsyntaxtextareaTokentype="${rsyntaxtextareaTokentypeRaw%"${rsyntaxtextareaTokentypeRaw##*[! ]}"}" + + # Reset IFS to default + unset IFS + + allowedKeywordCharactersRegex='^[a-zA-Z0-9_]+$' + # Check for corruption of KEYWORD field caused by UTF-8 BOM file encoding + if grep --quiet $'\xEF\xBB\xBF' <<<"$keyword"; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: UTF-8 BOM file encoding has corrupted the first keyword definition. Please change the file encoding to standard UTF-8." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_BOM_CORRUPTED_KEYWORD_EXIT_STATUS) + elif ! [[ "$keyword" =~ $allowedKeywordCharactersRegex ]]; then + # Check for invalid characters in KEYWORD + # The Arduino IDE does recognize keywords that start with a number, even though these are not valid identifiers. + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Keyword: $keyword contains invalid character(s), which causes it to not be recognized by the Arduino IDE. Keywords may only contain the characters a-z, A-Z, 0-9, and _." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_KEYWORD_EXIT_STATUS) + fi + + # Check for invalid KEYWORD_TOKENTYPE + local validKeywordTokentypeRegex='^((KEYWORD1)|(KEYWORD2)|(KEYWORD3)|(LITERAL1)|(LITERAL2))$' + local validRsyntaxtextareaTokentypeRegex='^((RESERVED_WORD)|(RESERVED_WORD_2)|(DATA_TYPE)|(PREPROCESSOR)|(LITERAL_BOOLEAN))$' + if ! [[ "$keywordTokentype" =~ $validKeywordTokentypeRegex ]]; then + # Check if it's invalid only because of leading space + local keywordTokentypeWithoutLeadingSpace="${keywordTokentype#"${keywordTokentype%%[![:space:]]*}"}" + if [[ "$keywordTokentypeWithoutLeadingSpace" =~ $validKeywordTokentypeRegex ]]; then + # Check if the issue doesn't cause any change from the intended highlighting + local inconsequentialTokentypeRegex='((KEYWORD2)|(KEYWORD3)|(LITERAL2))' + if [[ "$keywordTokentypeWithoutLeadingSpace" =~ $inconsequentialTokentypeRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Leading space on the KEYWORD_TOKENTYPE field causes it to not be recognized, so the default keyword highlighting is used. In this case that doesn't cause the keywords to be colored other than intended but it's recommended to fully comply with the Arduino library specification." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INCONSEQUENTIAL_LEADING_SPACE_ON_KEYWORD_TOKENTYPE_EXIT_STATUS) + else + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Leading space on the KEYWORD_TOKENTYPE field causes it to not be recognized, so the default keyword highlighting is used." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_LEADING_SPACE_ON_KEYWORD_TOKENTYPE_EXIT_STATUS) + fi + elif ! [[ "$keywordTokentypeWithoutLeadingSpace" == "" && "$rsyntaxtextareaTokentype" =~ $validRsyntaxtextareaTokentypeRegex ]]; then + # It's reasonable to leave KEYWORD_TOKENTYPE blank if RSYNTAXTEXTAREA_TOKENTYPE is defined and valid. This will not be compatible with 1.6.4 and older but that's really no big deal. + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Invalid KEYWORD_TOKENTYPE: $keywordTokentype causes the default keyword highlighting to be used. See: https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#keyword_tokentype" + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_KEYWORD_TOKENTYPE_EXIT_STATUS) + fi + fi + + # Check for invalid REFERENCE_LINK + if [[ "$referenceLink" != "" ]]; then + # The Arduino IDE must be installed to check if the reference page exists + if [[ "$NEWEST_INSTALLED_IDE_VERSION" == "" ]]; then + echo "WARNING: Arduino IDE is not installed so unable to check for invalid reference links. Please call install_ide before running check_keywords_txt." + else + install_ide_version "$NEWEST_INSTALLED_IDE_VERSION" + if [[ ! $(find "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/${ARDUINO_CI_SCRIPT_IDE_INSTALLATION_FOLDER}/reference/www.arduino.cc/en/Reference/" -type f -name "${referenceLink}.html") ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: REFERENCE_LINK value: $referenceLink is not a valid Arduino Language Reference page. See: https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#reference_link" + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_REFERENCE_LINK_EXIT_STATUS) + fi + fi + fi + + # Check for invalid RSYNTAXTEXTAREA_TOKENTYPE + if [[ "$rsyntaxtextareaTokentype" != "" ]]; then + if ! [[ "$rsyntaxtextareaTokentype" =~ $validRsyntaxtextareaTokentypeRegex ]]; then + # Check if it's invalid only because of leading space + local rsyntaxtextareaTokentypeWithoutLeadingSpace="${rsyntaxtextareaTokentype#"${rsyntaxtextareaTokentype%%[![:space:]]*}"}" + if [[ "$rsyntaxtextareaTokentypeWithoutLeadingSpace" =~ $validRsyntaxtextareaTokentypeRegex ]]; then + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Leading space on the RSYNTAXTEXTAREA_TOKENTYPE field causes it to not be recognized, so the default keyword highlighting is used." + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_LEADING_SPACE_ON_RSYNTAXTEXTAREA_TOKENTYPE_EXIT_STATUS) + else + echo "ERROR: ${normalizedKeywordsTxtPath}/keywords.txt: Invalid RSYNTAXTEXTAREA_TOKENTYPE: $rsyntaxtextareaTokentype causes the default keyword highlighting to be used. See: https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#rsyntaxtextarea_tokentype" + echo -e "\t$keywordsTxtLine" + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_KEYWORDS_TXT_INVALID_RSYNTAXTEXTAREA_TOKENTYPE_EXIT_STATUS) + fi + fi + fi + done <<<"$keywordsTxtCRline" + done <"${normalizedKeywordsTxtPath}/keywords.txt" + + done <<<"$(find "$normalizedKeywordsTxtSearchPath" -maxdepth "$maximumSearchDepth" -type d | sort --dictionary-order)" + + return "$exitStatus" +} + +# https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ#how-is-the-library-list-generated +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=1 +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_FOLDER_DOESNT_EXIST_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_EXE_FOUND_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_DOT_DEVELOPMENT_FOUND_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_SYMLINK_FOUND_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_CHECK_FOLDER_NAME_OFFSET=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_HAS_INVALID_FIRST_CHARACTER_EXIT_STATUS=$((ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_CHECK_FOLDER_NAME_OFFSET + ARDUINO_CI_SCRIPT_CHECK_FOLDER_NAME_INVALID_FIRST_CHARACTER_EXIT_STATUS)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_HAS_INVALID_CHARACTER_EXIT_STATUS=$((ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_CHECK_FOLDER_NAME_OFFSET + ARDUINO_CI_SCRIPT_CHECK_FOLDER_NAME_INVALID_CHARACTER_EXIT_STATUS)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_TOO_LONG_EXIT_STATUS=$((ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_CHECK_FOLDER_NAME_OFFSET + ARDUINO_CI_SCRIPT_CHECK_FOLDER_NAME_TOO_LONG_EXIT_STATUS)) +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_TOO_LONG_EXIT_STATUS + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_BLANK_URL_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER=$((ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER + 1)) +readonly ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_IS_RESERVED_EXIT_STATUS=$ARDUINO_CI_SCRIPT_EXIT_STATUS_COUNTER +function check_library_manager_compliance() { + local -r libraryPath="$1" + # Replace backslashes with slashes + local -r libraryPathWithSlashes="${libraryPath//\\//}" + # Remove trailing slash + local -r normalizedLibraryPath="${libraryPathWithSlashes%/}" + + local exitStatus=$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS + + # Check whether folder exists + if [[ ! -d "$normalizedLibraryPath" ]]; then + echo "ERROR: ${libraryPath}: Folder doesn't exist." + return $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_FOLDER_DOESNT_EXIST_EXIT_STATUS + fi + + # Check for .exe files + local -r exePath=$(find "$normalizedLibraryPath" -type f -name '*.exe') + if [[ "$exePath" != "" ]]; then + echo "ERROR: ${exePath}: .exe file found." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_EXE_FOUND_EXIT_STATUS) + fi + + # Check for .development file + local -r dotDevelopmentPath=$(find "$normalizedLibraryPath" -maxdepth 1 -type f -name '.development') + if [[ "$dotDevelopmentPath" != "" ]]; then + echo "ERROR: ${dotDevelopmentPath}: .development file found." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_DOT_DEVELOPMENT_FOUND_EXIT_STATUS) + fi + + # Check for symlink + local -r symlinkPath=$(find "$normalizedLibraryPath" -type l) + if [[ "$symlinkPath" != "" ]]; then + echo "ERROR: ${symlinkPath}: Symlink found." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_SYMLINK_FOUND_EXIT_STATUS) + fi + + # Check for problems with library.properties + if [[ -f "$normalizedLibraryPath/library.properties" ]]; then + # Get rid of the CRs + local libraryProperties + libraryProperties=$(tr "\r" "\n" <"$normalizedLibraryPath/library.properties") + local nameValue + nameValue="$(get_library_properties_field_value "$libraryProperties" 'name')" + + # Check if the library.properties name value meets the requirements of the Library Manager indexer + check_library_properties_name "$nameValue" + local -r checkLibraryPropertiesNameExitStatus=$? + if [[ $checkLibraryPropertiesNameExitStatus -ne $ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS ]]; then + if [[ "$checkLibraryPropertiesNameExitStatus" == "$ARDUINO_CI_SCRIPT_CHECK_LIBRARY_PROPERTIES_NAME_RESERVED_NAME_EXIT_STATUS" ]]; then + echo "ERROR: ${normalizedLibraryPath}/library.properties: name value: $nameValue starts with \"arduino\". These names are reserved for official Arduino libraries." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_NAME_IS_RESERVED_EXIT_STATUS) + else + echo "ERROR: ${normalizedLibraryPath}/library.properties: name value: $nameValue does not meet the requirements of the Arduino Library Manager indexer. See: https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format" + exitStatus=$(set_exit_status "$exitStatus" $((ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_CHECK_FOLDER_NAME_OFFSET + checkLibraryPropertiesNameExitStatus))) + fi + fi + + local urlValue + urlValue="$(get_library_properties_field_value "$libraryProperties" 'url')" + if [[ "$urlValue" == "" ]]; then + echo "ERROR: ${normalizedLibraryPath}/library.properties: Blank url value: You must define a URL." + exitStatus=$(set_exit_status "$exitStatus" $ARDUINO_CI_SCRIPT_CHECK_LIBRARY_MANAGER_COMPLIANCE_BLANK_URL_EXIT_STATUS) + fi + fi + + return "$exitStatus" +} + +function check_code_formatting() { + local -r strictness="$1" + local -r excludedPathList="$2" + local -r targetPath="$3" + + local -r astyleConfigurationFolder="etc/astyle-configurations" + local -r astyleConfigurationExtension=".conf" + + # Fold the output in the Travis CI log + echo -e -n 'travis_fold:start:check_code_formatting\r' + + if [[ $strictness -lt 1 ]] || [[ $strictness -gt 3 ]]; then + echo "ERROR: Invalid strictness parameter value. Valid values are 1 (least strict) - 3 (most strict)" + return "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi + + if ! [[ -d "$targetPath" ]]; then + echo "ERROR: targetPath doesn't exist" + return "$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi + + local scriptFolder + # https://stackoverflow.com/a/246128 + local scriptSource="${BASH_SOURCE[0]}" + while [ -h "$scriptSource" ]; do # Resolve $scriptSource until the file is no longer a symlink + scriptFolder="$(cd -P "$(dirname "$scriptSource")" >/dev/null 2>&1 && pwd)" + scriptSource="$(readlink "$scriptSource")" + [[ $scriptSource != /* ]] && scriptSource="$scriptFolder/$scriptSource" # If $scriptSource was a relative symlink, we need to resolve it relative to the path where the symlink file was located + done + scriptFolder="$(cd -P "$(dirname "$scriptSource")" >/dev/null 2>&1 && pwd)" + + # Assemble the find options for the excluded paths from the list + for excludedPath in ${excludedPathList//,/ }; do + excludeOptions="$excludeOptions -path $excludedPath -prune -or" + done + + local astylePath + astylePath=$(command -v astyle) + if [[ ! -e "$astylePath" ]]; then + # Install astyle + # Save the current folder + local -r previousFolder="$PWD" + mkdir --parents "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/astyle" + wget --no-verbose $ARDUINO_CI_SCRIPT_QUIET_OPTION --output-document="${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/astyle.tar.gz" "https://iweb.dl.sourceforge.net/project/astyle/astyle/astyle%203.1/astyle_3.1_linux.tar.gz" + tar --extract --file="${ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER}/astyle.tar.gz" --directory="${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}" + cd "${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/astyle/build/gcc" + eval "make $ARDUINO_CI_SCRIPT_VERBOSITY_REDIRECT" + astylePath="${ARDUINO_CI_SCRIPT_APPLICATION_FOLDER}/astyle/build/gcc/bin/astyle" + # Return to the previous folder + cd "$previousFolder" + fi + + # Set default exit status + exitStatus="$ARDUINO_CI_SCRIPT_SUCCESS_EXIT_STATUS" + + while read -r filename; do + # Check if it's a file (find matches on pruned folders) + if [[ -f "$filename" ]]; then + if ! diff --strip-trailing-cr "$filename" <("${astylePath}" --options="${scriptFolder}/${astyleConfigurationFolder}/${strictness}${astyleConfigurationExtension}" --dry-run <"$filename"); then + echo "ERROR: Non-compliant code formatting in $filename" + # Make the function fail + exitStatus="$ARDUINO_CI_SCRIPT_FAILURE_EXIT_STATUS" + fi + fi + done <<<"$(eval "find $targetPath -regextype posix-extended $excludeOptions \( -iregex '.*\.((ino)|(h)|(hpp)|(hh)|(hxx)|(h\+\+)|(cpp)|(cc)|(cxx)|(c\+\+)|(cp)|(c)|(ipp)|(ii)|(ixx)|(inl)|(tpp)|(txx)|(tpl))$' -and -type f \)")" + + echo -e -n 'travis_fold:end:check_code_formatting\r' + return "$exitStatus" +} # Set default verbosity (must be called after the function definitions set_script_verbosity 0 - # Create the temporary folder +rm "$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" --recursive --force create_folder "$ARDUINO_CI_SCRIPT_TEMPORARY_FOLDER" # Create the report folder create_folder "$ARDUINO_CI_SCRIPT_REPORT_FOLDER" - # Add column names to report -echo "Build Timestamp (UTC)"$'\t'"Build"$'\t'"Job"$'\t'"Job URL"$'\t'"Build Trigger"$'\t'"Allow Job Failure"$'\t'"PR#"$'\t'"Branch"$'\t'"Commit"$'\t'"Commit Range"$'\t'"Commit Message"$'\t'"Sketch Filename"$'\t'"Board ID"$'\t'"IDE Version"$'\t'"Program Storage (bytes)"$'\t'"Dynamic Memory (bytes)"$'\t'"# Warnings"$'\t'"Allow Failure"$'\t'"Exit Status"$'\t'"# Board Issues"$'\t'"Board Issue"$'\t'"# Library Issues"$'\t'"Library Issue"$'\r' > "$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" - - -# Start the virtual display required by the Arduino IDE CLI: https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#bugs -# based on https://learn.adafruit.com/continuous-integration-arduino-and-you/testing-your-project -/sbin/start-stop-daemon --start $ARDUINO_CI_SCRIPT_QUIET_OPTION $ARDUINO_CI_SCRIPT_VERBOSITY_OPTION --pidfile /tmp/custom_xvfb_1.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :1 -ac -screen 0 1280x1024x16 -sleep 3 -export DISPLAY=:1.0 +echo "Build Timestamp (UTC)"$'\t'"Build"$'\t'"Job"$'\t'"Job URL"$'\t'"Build Trigger"$'\t'"Allow Job Failure"$'\t'"PR#"$'\t'"Branch"$'\t'"Commit"$'\t'"Commit Range"$'\t'"Commit Message"$'\t'"Sketch Filename"$'\t'"Board ID"$'\t'"IDE Version"$'\t'"Program Storage (bytes)"$'\t'"Dynamic Memory (bytes)"$'\t'"# Warnings"$'\t'"Allow Failure"$'\t'"Exit Status"$'\t'"# Board Issues"$'\t'"Board Issue"$'\t'"# Library Issues"$'\t'"Library Issue"$'\r' >"$ARDUINO_CI_SCRIPT_REPORT_FILE_PATH" diff --git a/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/1.conf b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/1.conf new file mode 100755 index 00000000..fe9cf302 --- /dev/null +++ b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/1.conf @@ -0,0 +1,33 @@ +# source: https://raw.githubusercontent.com/arduino/Arduino/1.8.9/build/shared/lib/formatter.conf + +# This configuration file contains a selection of the available options provided by the formatting tool "Artistic Style" +# http://astyle.sourceforge.net/astyle.html +# +# If you wish to change them, don't edit this file. +# Instead, copy it in the same folder of file "preferences.txt" and modify the copy. This way, you won't lose your custom formatter settings when upgrading the IDE +# If you don't know where file preferences.txt is stored, open the IDE, File -> Preferences and you'll find a link + +mode=c + +# 2 spaces indentation +indent=spaces=2 + +# also indent macros +indent-preprocessor + +# indent classes, switches (and cases), comments starting at column 1 +indent-classes +indent-switches +indent-cases +indent-col1-comments + +# put a space around operators +pad-oper + +# put a space after if/for/while +pad-header + +# if you like one-liners, keep them +keep-one-line-statements + +remove-comment-prefix diff --git a/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/2.conf b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/2.conf new file mode 100755 index 00000000..124f5474 --- /dev/null +++ b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/2.conf @@ -0,0 +1,46 @@ +# source: https://raw.githubusercontent.com/arduino/Arduino/1.8.9/build/shared/examples_formatter.conf + +# This configuration file contains a selection of the available options provided by the formatting tool "Artistic Style" +# http://astyle.sourceforge.net/astyle.html +# +# If you wish to change them, don't edit this file. +# Instead, copy it in the same folder of file "preferences.txt" and modify the copy. This way, you won't lose your custom formatter settings when upgrading the IDE +# If you don't know where file preferences.txt is stored, open the IDE, File -> Preferences and you'll find a link + +mode=c + +# 2 spaces indentation +indent=spaces=2 + +# also indent macros +indent-preprocessor + +# indent classes, switches (and cases), comments starting at column 1 +indent-classes +indent-switches +indent-cases +indent-col1-comments + +# put a space around operators +pad-oper + +# put a space after if/for/while +pad-header + +# if you like one-liners, keep them +keep-one-line-statements + +style=java +attach-namespaces +attach-classes +attach-inlines +attach-extern-c +indent-modifiers +indent-namespaces +indent-labels +indent-preproc-block +indent-preproc-define +indent-preproc-cond +unpad-paren +add-brackets +remove-comment-prefix diff --git a/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/3.conf b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/3.conf new file mode 100755 index 00000000..66613c27 --- /dev/null +++ b/avr/travis-ci/arduino-ci-script/etc/astyle-configurations/3.conf @@ -0,0 +1,125 @@ +# source: https://github.com/arduino/ArduinoCore-avr/issues/71#issuecomment-466763471 2019-06-09 + +# formatter.conf, examples_formatter.conf +mode=c + + +# examples_formatter.conf +# http://astyle.sourceforge.net/astyle.html#_style=java +# Considering changing this to the synonym "style=attach", which seems more descriptive +style=java + + +# examples_formatter.conf +attach-namespaces + +# examples_formatter.conf +attach-classes + +# examples_formatter.conf +attach-inlines + +# examples_formatter.conf +attach-extern-c + + +# formatter.conf, examples_formatter.conf +indent=spaces=2 + +# formatter.conf, examples_formatter.conf +indent-classes + +# formatter.conf, examples_formatter.conf +indent-switches + +# formatter.conf, examples_formatter.conf +indent-cases + +# formatter.conf, examples_formatter.conf +indent-col1-comments + +# examples_formatter.conf +indent-modifiers + +# examples_formatter.conf +indent-namespaces + +# examples_formatter.conf +indent-labels + +# examples_formatter.conf +indent-preproc-define + + +# formatter.conf, examples_formatter.conf +pad-header + +# formatter.conf, examples_formatter.conf +pad-oper + +# examples_formatter.conf +unpad-paren + + +# formatter.conf, examples_formatter.conf +remove-comment-prefix + +# formatter.conf, examples_formatter.conf +# http://astyle.sourceforge.net/astyle.html#_keep-one-line-statements +# "Don't break complex statements and multiple statements residing on a single line." +# I don't like one line complex statements, but I guess since it's in formatter.conf it must stay. +keep-one-line-statements + + + +# Options from examples_formatter.conf that I think should be removed: + +# http://astyle.sourceforge.net/astyle.html#_indent-preproc-block +# "Indent preprocessor blocks at brace level zero and immediately within a namespace. There are restrictions on what will be indented. Blocks within methods, classes, arrays, etc., will not be indented. Blocks containing braces or multi-line define statements will not be indented. Without this option the preprocessor block is not indented." +# This does indent for #ifdef, but not for #ifndef, so it's quite inconsistent +# Indentation of preprocessor directives as done by this option is not very common in Arduino AVR Boards core, and where it is used, it's typically done inconsistently throughout the file +indent-preproc-block + +# http://astyle.sourceforge.net/astyle.html#_indent-preproc-cond +# "Indent preprocessor conditional statements to the same level as the source code." +# Indentation of preprocessor directives as done by this option is very rare in Arduino AVR Boards core +indent-preproc-cond + + + +# Options I have not implemented from formatter.conf or examples_formatter.conf: + +# examples_formatter.conf +# Not a valid option in the latest version of AStyle. I think the correct option name is "add-braces", which I do use in my configuration +# add-brackets + +# formatter.conf, examples_formatter.conf +# Not a valid option in the latest version of AStyle. +# indent-preprocessor + + + +# Options I have added: + +# http://astyle.sourceforge.net/astyle.html#_add-braces +# "I believe this is the correct option name to use instead the "add-brackets" option used in examples_formatter.conf. "add-brackets" is not a valid option in the latest version of AStyle" +add-braces + +# http://astyle.sourceforge.net/astyle.html#_convert-tabs +# "Converts tabs into spaces in the non-indentation part of the line." +# AStyle is already configured to use spaces for indentation by indent=spaces=2. The Arduino IDE uses spaces instead of tabs by default. +convert-tabs + +# http://astyle.sourceforge.net/astyle.html#_attach-return-type +# "Attach the return type to the function name in function definitions." +attach-return-type + +# http://astyle.sourceforge.net/astyle.html#_attach-return-type +# "Attach the return type to the function name in function declarations." +attach-return-type-decl + +# http://astyle.sourceforge.net/astyle.html#_align-pointer +# "Attach a pointer or reference operator (*, &, or ^) to either the variable type (left) or variable name (right), or place it between the type and name (middle)." +# In Arduino AVR Boards core, name alignment of pointers is somewhat more common, though all possible styles are used +# I don't care which style is chosen (type, middle, name), but I do think one should be chosen and used. +align-pointer=name diff --git a/avr/travis-ci/arduino-ci-script/etc/autoformat.sh b/avr/travis-ci/arduino-ci-script/etc/autoformat.sh new file mode 100755 index 00000000..0ef37d9b --- /dev/null +++ b/avr/travis-ci/arduino-ci-script/etc/autoformat.sh @@ -0,0 +1 @@ +shfmt -i 2 -w ../arduino-ci-script.sh diff --git a/avr/travis-ci/arduino-ci-script/etc/codespell-ignore-words-list.txt b/avr/travis-ci/arduino-ci-script/etc/codespell-ignore-words-list.txt new file mode 100755 index 00000000..c74fb046 --- /dev/null +++ b/avr/travis-ci/arduino-ci-script/etc/codespell-ignore-words-list.txt @@ -0,0 +1,4 @@ +ba +propert +te +exampels