On Gulp

– Starting up:

$ npm install –global gulp

$ npm install –save gulp

– Create a gulpfile.js on your root directory, sample content:

var gulp = require(‘gulp’);

gulp.task(‘welcome’, function () {

  console.log(‘gulp is working!’);

});

– run your gulp functions / commands:

$ gulp welcome

note: some linux installations refer to nodejs as “node” internally. If you get the following error message when running your gulp commands:

/usr/bin/env: node: No such file or directory

that means you need to create a soft link to alias node to nodejs:

ln -s /usr/bin/nodejs /usr/bin/node

– install the pluging that will let you concat your js files together in one big file:

$ npm install –save gulp-concat

– install the plugin that will compact and uglify your js (needs to go along with the other one, to make uglification compatible with angular)

npm install –save gulp-uglify

$ npm install –save gulp-ng-annotate

– here is an example that will concat and uglify into an app.js file, and put it into your assets directory, loading file ng/module.js first in the resulting file:

var gulp = require(‘gulp’);

var concat = require(‘gulp-concat’);

var uglify = require(‘gulp-uglify’);

var ngAnnotate = require(‘gulp-ng-annotate’);

gulp.task(‘js’, function () {

  gulp.src([‘ng/module.js’, ‘ng/**/*.js’])

  .pipe(concat(‘app.js’))

  .pipe(ngAnnotate())

  .pipe(uglify())

  .pipe(gulp.dest(‘assets’));

});

 

boostrapping a MEAN app (angular.js, express, node.js, mongodb)

Complete code example here

– in the folder you are creating your app, place a manifest file called package.json, example:

{
“name”: “yourappnamehere”
}

– install express and other dependencies:

$ sudo apt-get update

$ sudo apt-get install npm

sudo apt-get install nodejs

Note: copying and pasting these commands sometimes will give you the following error message:

TypeError: Cannot read property ‘latest’ of undefined

make sure to retype the “–” part manually, and you’ll be all right

$ sudo npm install –save express

$ sudo npm install –save body-parser

– if you want node to automatically restart when there are changes in the files, you can also install the following package:

$ npm install –global nodemon

– and then, when you start your server, you need to start it as:

$ npm install –global nodemon

– install mongodb, create the following file on your home directory: mongo_install.bash, with the following content:

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" | tee -a /etc/apt/sources.list.d/10gen.list
apt-get -y update
apt-get -y install mongodb-10gen

– run sudo bash ./mongo_install.bash

– the installation starts mongod by default, but that is the daemon you need to start if you don’t see it running

– install mongoose:

$ npm install –save mongoose

– to enter the console mode (and verify the installation), type mongo, if you want to connect to an specific db, you do:

mongo nameofyourdbhere

> db.posts.find()

that will give you all the records saved under the Post model

– create a server.js file, that will host your app (see code in heroku instance for content details)

– run your server:

nodejs server.js

(config.vm.network :forwarded_port, guest: 3000, host: 3000 on Vagrantfile if you are running inside vagrant)

– server.js is kind of your single point of entry for your app. It is always a good idea to keep it lean, and move as much code as possible away from it into other files. Some things that are worth having at this file are:

— the server listening loop

— global configuration and other middleware packages

— logging and error handling

— controllers spawning and mounting

– on static files: it is a good idea not to serve them via nodejs. Try to keep your node instance as an API, and let apache and other cache services to do the static servers job. But if you must, it is always a good idea to spin them into:

/controllers/static.js

and inside that file:

var express = require(‘express’)
var router = express.Router()

router.use(express.static(__dirname + ‘/../assets’))

– so now any file you put on your /assets folder will be served by node

– on services: things like $http are better constructed via a service, and then injected to wherever they are needed. Below is an example of doing just that:

app.service(‘PostsSvc’, function ($http) {

  this.fetch = function () {

  return $http.get(‘/api/posts’)

  }

  this.create = function (post) {

  return $http.post(‘/api/posts’, post)

  }

});

– and then, the controllers that consume it would looks something like this:

  // create the PostsCtrl module

  // dependency inject $scope

  app.controller(‘PostsCtrl’, function ($scope, PostsSvc) {

  // the function runs when the “Add Post” button is clicked

$scope.addPost = function () {

  if ($scope.postBody) {

  PostsSvc.create({

  username: ‘ramiro’,

  body: $scope.postBody

  }).success(function (post) {

  $scope.posts.unshift(post)

  $scope.postBody = null

  })

  }

  };

 Deploying to Heroku

– create a .gitignore file with the following lines:

node_modules
assets

– if you are in a new vagrant instance, install heroku tools first:

$ wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh

$ heroku login

heroku create your-app-name-here

heroku addons:create mongolab

– check what is the address of your mongolabs instance:

$ heroku config

– modify your db.js file according to what you see printed by that command, it would look something like this:

var mongoose = require(‘mongoose’);

var url = process.env.MONGOLAB_URI || ‘mongodb://localhost/social’;

mongoose.connect(url);

module.exports = mongoose;

– you also need to do a similar move for the listen command in the server.js file:

// process.env.PORT for the benefit of Heroku

app.listen(process.env.PORT || 3000, function () {

  console.log(‘Server listening on’, 3000)

});

bootstapping an angular.js app on top of rails

Complete example here

– The following will skip jquery and turbolinks:

rails new [your-app-name-here] –skip-javascript

– But now you need to manually create the following file: app/assets/javascripts/application.js

– And include the following in the file contents:

//= require_tree .

– put all your angular app files inside app/assets/javascripts, and include the following in your app/views/layouts/application.html.erb template:

<%= javascript_include_tag ‘application’ %>

– Your app/views/layouts/application.html.erb will also be the home page of your single page app, so make sure you put all your initial HTML code in there, and get rid of the default <%= yield %> in there.

– Add the following to your application controller:

def angular

render ‘layouts/application’

end

– and in your config/ routes.rb file, add a route to it:

root to: ‘application#angular’

– install bower, if you haven’t done so already:

npm install -g bower

– initialize it inside your rails project:

bower init

– create a .bowerrc file, with the following, to tell bower where you are storing your js dependencies:

{ “directory“:“vendor/assets/bower_components” }

– install the dependencies you need, and tell bower to save them in your config file:

bower install angular angular-ui-router bootstrap –save

– now that the libraries you need are installed, modify your app/assets/javascripts/application.js file to call them at page load time:

//= require angular

//= require angular-ui-router

//= require_tree .

– to add the bootstrap CSS, add the following line to the comments section on app/assets/stylesheets/application.css

*= require bootstrap/dist/css/bootstrap

*= require_tree .

Important: that goes inside the header /* comments */

At this point, you have all your js / css dependencies managed by bower, and minified and pulled in the right places by the rails app.

– place your template files inside public/templates, and modify your config file as follows:

.config([

'$stateProvider',
'$urlRouterProvider',
function($stateProvider, $urlRouterProvider) {

$stateProvider
.state(‘home’, {
url: ‘/home’,
templateUrl: ‘templates/_home.html’,
controller: ‘MainCtrl’
}).state(‘posts’, {
url: ‘/posts/{id}’,
templateUrl: ‘templates/_posts.html’,
controller: ‘PostsCtrl’
});

$urlRouterProvider.otherwise(‘home’);
}])

– move your JS to folders inside app/assets/javascripts (create mainCtrl.js inside the home folder, and postsCtrl.js inside the posts folder), so the only thing left inside app.js is the configuration and routes.

– generate your models:

rails generate model Post title:string link:string upvotes:integer

rails generate model Comment body:string upvotes:integer post:references

rake db:migrate

– declare your associations inside your models:

class Post < ActiveRecord::Base

  has_many :comments

end

– if you have children models from one of your models, and you want the children to be returned as part of your JSON, put the following method inside your model as well:

def as_json(options = {})

 super(options.merge(include: :comments))

end

 – setup your base routes:

root to: ‘application#angular’

resources :posts, only: [:create, :index, :show] do

 resources :comments, only: [:show, :create] do

 member do put ‘/upvote’ => ‘comments#upvote’

 end

end

member do put ‘/upvote’ => ‘posts#upvote’ end end

Nesting resources will create urls like this: posts/1/comment/3

By specifying member do, you are also simplifying some of the resources out of the deep nesting

 – using “rake routes” at this point will tell you what kind of routes you have defined

– now is time to create your controllers, make sure you skip the templates and assets creation when doing so:

rails generate controller Posts –skip-assets –skip-template-engine

rails generate controller Comments –skip-assets –skip-template-engine

 – add the following line to your application controller, so it responds back in json format:

respond_to :json

– in the newer versions of rails, you will need to add this gem to your Gemfile to get that respond_to functionality:

gem ‘responders’, ‘~> 2.0’

 – tell your controllers which parameters are allowed, and also create the basic controller methods:

def index
respond_with Post.all
end

def create
respond_with Post.create(post_params)
end

def show
respond_with Post.find(params[:id])
end

def upvote
post = Post.find(params[:id])
post.increment!(:upvotes)

respond_with post
end

private
def post_params
params.require(:post).permit(:link, :title)
end
end

– back to your front end: setup your factory to be able to retrieve all of your records by means of hitting your index method (see how $http is injected, and how the getAll function is implemented as a promise)

angular.module(‘flapperNews’)

.factory(‘posts’, [‘$http’, function(){

  var o = {

  posts: []

  };

  o.getAll = function() {

  return $http.get(‘/posts.json’).success(function(data){

  angular.copy(data, o.posts);

  });

  };

  return o;

}]);

– now, if you want your view to refresh with the server data every time the UI calls “home”, you need to set your stateProvider with the “resolve” property:

$stateProvider

  .state(‘home’, {

  url: ‘/home’,

  templateUrl: ‘templates/_home.html’,

  controller: ‘MainCtrl’,

  resolve: {

  postPromise: [‘posts’, function(posts){

 return posts.getAll();

 }]

  }

  })

 – to be able to create post, add the following to your post service:

o.create = function(post) { return $http.post(‘/posts.json’, post).success(function(data){ o.posts.push(data); }); };

– and, in your main controller:

$scope.addPost = function(){ if(!$scope.title || $scope.title === ) { return; } posts.create({ title: $scope.title, link: $scope.link, }); $scope.title = ; $scope.link = ; };

– by default, rails has protection against fake posts, so in order to save the data you are posting, you will need to add the following gem (otherwise you will be getting “422 unprocessable entry” error messages

gem ‘angular_rails_csrf’

– to add user authentication via Device, you need to install the gem first:

gem ‘devise’, ‘~> 3.4.0’

– after you bundle install, initialize it, and create the user’s model:

rails generate devise:install

rails generate devise User

– if you need to, you can add more fields to the default devise model, which only contain email and password by default. We will also make the username unique:

rails generate migration AddUsernameToUser username:string:uniq

– in order to integrate devise with the front end, you can install the following js package helper via bower:

bower install angular-devise –save

– in application.js, require the newly installed package:

//= require angular-devise

– and inject the module in the main app:

angular.module(‘flapperNews’, [‘ui.router’, ‘Devise’]).

# note: when you try your registration / login forms, if there is a devise error, it may manifest in the front end as a 422 error message, you need to handle the errors as they come from the server

– to secure your posts savings (or any other controller actions for that matter) you can now use the following:

class PostsController < ApplicationController before_filter :authenticate_user!, only: [:create, :upvote]

– if you want to associate two of the models you are working with (in this case posts and users), run the following command, that will create the db migration necessary to do the work:

rails g migration AddUserRefToPosts user:references

rails g migration AddUserRefToComments user:references

rake db:migrate

– if you do so, you need to enhance the models to reflect that association:

class Comment < ActiveRecord::Base

  belongs_to :user

– and your update and create methods also need to include the relationship, so rails knows at save time what users are assigned what records:

def create

respond_with Post.create(post_params.merge(user_id: current_user.id))

end

vagrant: the basics

Dependencies: VirtualBox or VMware (virtualization software)

# To add new boxes you can run locally:

vagrant box add <the name of the box here>

# the list of boxes at:

https://atlas.hashicorp.com/boxes/search

vagrant box list    # gives you the list of boxes locally available

# to use one of those:

cd <the directory you want to work with vagrant>

vagrant init

vagrant init ubuntu/trusty64  # ubuntu/trusty64 is the name of your local box

vagrant up # to start the virtual machine

vagrant suspend   # put your machine to sleep, all save to disk

vagrant resume   # bring it back up

vagrant halt     # turn off your virtual machine, but things will still be kept in disk for later use

vagrant destroy  # machine is annihilated, but the files are still kept in disk, deeper hibernation

vagrant reload # halth and up equivalent

vagrant ssh  #ssh to the virtual machine

cd /vagrant/  # this is a shared folder between your local computer and your virtual machine

Inside your Vagrantfile configuration

# remember: this file must be inside your root folder for your VM

# to make a server running on port 80 in your vm accessible from outside:

config.vm.network :forwarded_port, 80, host: 8080 

# to have shared files between your VM and your local machine:

config.vm.synced_folder “../data”, “/vagrant_data”

the /vagrant_data file will actually be located in the root folder of your virtual machine “/” (not your home directory)

# to run shell commands at the end of the machine coming up, comment out:config.vm.provision “shell” (either put your commands inline here, or do a :path to the file that contain the commands)

# you can also just run the provision part of vagrant by running:

vagrant provision

# to remove local boxes:

vagrant box remove <name of the box to remove>

# to use the same ssh keys as in the host machine (inside the vagrant instance):

config.ssh.forward_agent = true

vagrant and puppet

vagrant ssh; which puppet # to make sure you have puppet

# to install it (if missing):

apt-get install puppet

# to point your provisioning to puppet (inside your Vagrant file):

config.vm.provision :puppet

# this will make provisioning to look for the following file for configuration:

manifest/default.pp

vagrant and rails 

modify the following line (to port forward to the external world from your VM):
config.vm.network “forwarded_port”, guest: 3000, host: 3000

if you want to port forward using rails, remember to startup your rails app as the following:

rails s -b 0.0.0.0

The basic commands to get a rails app ready inside your Vagrant folder (and even throwing Heroku in the mix):

sudo apt-get update
sudo apt-get install curl
curl -L https://get.rvm.io | bash -s stable

# if the above fails, try:

gpg –keyserver hkp://keys.gnupg.net –recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

and:

command curl -sSL https://rvm.io/mpapis.asc | gpg –import –

and then command again if the first one fails
source ~/.rvm/scripts/rvm
rvm requirements
rvm install ruby
rvm use ruby –default
rvm rubygems current
gem install rails

rails -v

Note: if you have trouble running bundle install because of nokogiri, try the following:

$ sudo apt-get install libxml2-dev libxslt1-dev

Also, if you have trouble installing postgres (Can’t find the ‘libpq-fe.h header error) then try this before bundle install:

sudo apt-get install libpq-dev

If you end up using postgres for your local ubuntu dev environment, you will need to setup the vagrant user as a database creator:

$ sudo -i -u postgres

# CREATE ROLE vagrant LOGIN;

ALTER ROLE vagrant WITH CREATEDB;

# \q

Inside database.yml, don’t forget to specify postgres as your database of choice:

 adapter: postgresql

$ rake db:create

$ rake db:migrate

And finally, the heroku tools:

wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh

 

Vagrant and git

sudo apt-get install git

 

Vagrant and node.js

The following three commands will get you setup:

sudo apt-get update
sudo apt-get install nodejs
sudo apt-get install npm

Vagrant and bower (or any npm that don’t seem to run inside ubuntu)

sudo ln -s /usr/bin/nodejs /usr/bin/node

Vagrant and ionic

If you are having trouble making port forwarding work at “ionic serve” time, try this:

$ ionic serve –address 0.0.0.0

And also, remember to port forward both ports in your vagrant file (not only the server one):

  config.vm.network :forwarded_port, host: 8100, guest: 8100

  config.vm.network :forwarded_port, host: 35729, guest: 35729

Angularjs with Ruby On Rails app from scratch

# Assuming node is already installed, install the stuff needed to bootstrap the app:

npm install -g yo

npm install -g generator-angular

# create your app (the name of the folder will be the name of your app):

mkdir myapp; cd myapp;

yo angular

# in the series of questions, pick bootstrap (to make it easier on you)

# get git going:

git init; git add .

git commit -am “initial commit”

# (optional): open your Gruntfile.js file, and edit out the following line:

tasks: [‘newer:jshint:all’], and also tasks: [‘newer:jshint:test’, ‘karma’]

to:

tasks: [] # to avoid running jshint for each edit, and the test suite with any new test / changes

# bowel.json will contain all app dependencies, if you need anything more, you add it there

# package.json will contain all the development dependencies

# app/index.html has your single page app bootstrapped HTML page

# to make sure we are up and running, run:

grunt serve # you shall have a basic page running at this point

ETL Transforms: some basic techniques

Some basic principles:

  • Select your indexes and sortable columns wisely, it can be the difference between hours and seconds runs
  • Try to process your data in chunks, create a staging table where you do your transforms with smaller amounts of data, instead of transforming all the data at all times.
  • Try to batch the changes by timestamp, pick a timestamp that will make all the data consistent across all your tables

Some techniques to transform data (note that they are tailored made for redshift, may be different in other databases):

Replace characters at the same time you are selecting

SELECT 
 select id,
 user_id,
 value,
 replace(replace(metadata, 'n', ' '), 't', '') as clean_metadata
FROM some_table;

Augmenting data in one table, by adding columns from another table

 SELECT
 du.user_id,
 du.site_id,
 du.user_name,
 up.earned_credits as current_point_balance,
 up.balance as lifetime_point_balance
 FROM base_table du
 LEFT JOIN enhancer_table up
 ON (du.user_id = up.user_id)
 WHERE up.debits is not null OR up.balance is not null;

 

Query using the IN keyword, to see if the values exist in another subquery

 SELECT * 
 FROM table_name WHERE (user_id, updated) IN 
 ( SELECT user_id, MAX(updated)
 FROM add_points
 GROUP BY user_id
 )

 

Convert a timestamp into a day id

date_trunc('day', create_time)


datediff(day, '2005-01-01', your_timestamp_column) + 1


Breaking your query into more manageable subqueries

WITH add_day_id as (
 SELECT
 site_id,
 user_id,
 someothercolum
 FROM sometable
),

add_something_else as (
 SELECT
 (some other transform here)
 FROM add_day_id
)

SELECT * FROM add_something_else;

Using CASE and if statements to select column values

SELECT
 some_column,
 CASE
 WHEN day_id = 10000
 THEN -1
 ELSE
 coalesce(
 lead(day_id) 
 OVER (
 PARTITION BY
 site_id,
 user_id,
 point_category_id,
 day_id
 ORDER BY
 site_id, user_id, day_id
 ),-1)
 END AS last_record_in_day
 FROM some_table

Check if a record exists in some other table, and putting its existance as part of the select condition

SELECT something FROM somewhere
WHERE some_condition = 'something'
AND exists (
   SELECT 1
   FROM some_other_table ot
   WHERE ot.site_id = full.reference.to.this.table.site_id
   AND (same as above, as many times as you need)
);

 Creating (or replacing) temp views, also, a way to mark row numbers by a certain order, by creating an artificial column for that

CREATE OR REPLACE VIEW temp_some_view_name AS (
WITH
mark_rows as (
 SELECT
 site_id,
 user_id,
 start_ts,
 ROW_NUMBER() OVER(
 PARTITION BY
 site_id,
 user_id
 ORDER BY start_ts) AS row_num
FROM some_table
),
pick_only_first_recs_on_groups as (
   SELECT
      site_id,
      user_id,
      start_ts
FROM mark_rows WHERE row_num = 1
)
SELECT * from pick_only_first_recs_on_groups);

least, to choose the lessen of all values

select some_column, least((SELECT day_id from dim_day where actual_date  = date_trunc('day', start_ts)), some_default_value) as some_other_column, yet_another_column FROM * sometable;

Coalesce, to avoid nulls and set defaults

select some_column, coalesce((SELECT day_id from dim_day where actual_date  = date_trunc('day', start_ts)), 1) as some_other_column, yet_another_column FROM * sometable;

Ionic: the basics

$ ionic serve // Internal server to run and test your pages

$ ionic build ios

$ ionic emulate ios // with the other command above, emulates running the page in the iphone

To use it along with your Adobe Phonegap build account (so you don’t have to build locally)

  1. Follow the instructions here to setup your phonegap acct
  2. $phonegap create [your app dir] [your app name]
  3. $cd [your app dir]
  4. copy the www files from your ionic app into the www of this phonegap dir (follow the instructions on their website to build a starter project)
  5. $phonegap remote build ios

Update: looks like things have gotten easier, and you don’t have to jump back and forth between phonegap and ionic projects, all you have to do is:

1) create an app at https://apps.ionic.io/

2) in your command line:

$ ionic start myApp io

$ cd myApp

$ionic login

$ionic upload

Setting up sass

ionic setup sass

if you run into an error message, you may have to reinstall the following packages (in that order):

$ sudo npm install node-sass@1.0.3

sudo npm install gulp-sass