Sunday, May 28, 2017

Software Define Network


What’s Software Defined Networking (SDN)? Definition


Software Defined Networking (SDN) The goal of Software-Defined Networking is to enable cloud and network engineers and administrators to respond quickly to changing business requirements via a centralized control console.   SDN encompasses multiple kinds of network technologies designed to make the network more flexible and agile to support  the virtualized server and storage infrastructure of the modern data center and Software defined networking was originally defined an approach to designing, building, and managing networks that separates the network’s control (brains) and forwarding (muscle) planes enabling the network control to become directly programmable and the underlying infrastructure to be abstracted for applications and network services.

How Does Software-Defined Networking or SDN Work?

Software-defined networking providers offer a wide selection of competing architectures, but at its most simple, the Software Defined Networking method centralizes control of the network by separating the control logic to off-device computer resources. All SDN models have some version of an SDN Controller, as well as southbound APIs and northbound APIs:
  • Controllers: The “brains” of the network, SDN Controllers offer a centralized view of the overall network, and enable network administrators to dictate to the underlying systems (like switches and routers) how the forwarding plane should handle network traffic.
  • Southbound APIs: Software-defined networking uses southbound APIs to relay information to the switches and routers “below.” OpenFlow, considered the first standard in SDN, was the original southbound API and remains as one of the most common protocols. Despite some considering OpenFlow and SDN to be one in the same, OpenFlow is merely one piece of the bigger SDN landscape.
  • Northbound APIs: Software Defined Networking uses northbound APIs to communicates with the applications and business logic “above.” These help network administrators to programmatically shape traffic and deploy services.

The Software Defined Networking Framework

Software Defined Networking (SDN) Framework
Software Defined Networking (SDN) Framework

Software-Defined Networking is Not OpenFlow

Often people point to OpenFlow as being synonymous with software-defined networking, but it is only a single element in the overall SDN architecture. OpenFlow is an open standard for a communications protocol that enables the control plane to interact with the forwarding plane. It must be noted that OpenFlow is not the only protocol available or in development for SDN.

The Benefits of Software Defined Networking

Offering a centralized, programmable network that can dynamically provision so as to address the changing needs of businesses, software-define networking also provides the following benefits:
  • Directly Programable:  Network directly programmable because the control functions are decoupled from forwarding functions.which enable the network to be programmatically configured by proprietary or open source automation tools, including OpenStackPuppet, and Chef.
  • Centralized Management:  Network intelligence is logically centralized in SDN controller software that maintains a global view of the network, which appears to applications and policy engines as a single, logical switch.
    Reduce CapEx
    : Software Defined Networking potentially limits the need to purchase purpose-built, ASIC-based networking hardware, and instead supports pay-as-you-grow models
  • Reduce OpEX: SDN enables algorithmic control of the network of network elements (such as hardware or software switches / routers that are increasingly programmable, making it easier to design, deploy, manage, and scale networks. The ability to automate provisioning and orchestration optimizes service availability and reliability by reducing overall management time and the chance for human error.
  • Deliver Agility and Flexibility: Software Defined Networking helps organizations rapidly deploy new applications, services, and infrastructure to quickly meet changing business goals and objectives.
  • Enable Innovation: SDN enables organizations to create new types of applications, services, and business models that can offer new revenue streams and more value from the network.

Why Software Defined Networking Now?

Social media, mobile devices, and cloud computing are pushing traditional networks to their limits. Compute and storage have benefited from incredible innovations in virtualization and automation, but those benefits are constrained by limitations in the network. Administrators may spin up new compute and storage instances in minutes, only to be held up for weeks by rigid and oftentimes manual network operations.
Software-defined networking has the potential to revolutionize legacy data centers by providing a flexible way to control the network so it can function more like the virtualized versions of compute and storage today.

Software Defined Networking Use Cases

As detailed above, Software Defined Networking offers several benefits for businesses trying to move into a virtual environment. There are a multitude of use cases that SDN offers for different organizations, including carrier and service providers, cloud and data centers, as well as enterprise campuses.
For carrier and service providers, Software-Defined Networking offers bandwidth on demand, which gives controls on carrier links to request additional bandwidth when necessary, as well as WAN optimization and bandwidth calendaring. For cloud and data centers, network virtualization for multi-tenants is an important use case as it offers better utilization of resources and faster turnaround times for creating a segregated network. Enterprise campuses experience network access control and network monitoring when using Software-Defined Networking policies.
Please Note this article is from https://www.sdxcentral.com/sdn/definitions/what-the-definition-of-software-defined-networking-sdn/

From bare metal and virtual machines to Docker and Kubernetes

Kubernetes是一个分布式操作系统,它能调度很多Docker containers。分布式系统的核心在网络和通信。要理解Kubernetes如何处理网络,最好从分布式系统的网络演进入手,知其然知其所以然。

Bare Metal Network

最简单粗暴的分布式计算就是在几台机器(bare metal computer)上启动一些进程,让它们之间可以通信。这里有两个概念:
  1. 节点(node),和
  2. 进程(process)
其中节点也叫主机(host),因为进程是客人。
一个进程A要连上另一个进程B的时候,得知道B的网络地址。在TCP/IP网络里,每台机器有一个IP地址,同一台机器上不同的进程(线程)可以监听不同的端口(port),于是一个进程(线程)可以用一个IP地址加上一个port来定位,这就是网络地址(network address)。
  • 同一台机器上进程互联的时候可以用特殊的loopback IP 127.0.0.1。操作系统的网络驱动程序处理和127.0.0.1上的某个port的通信的时候,是不调用网卡驱动程序的,这样效率很高。下图中 process 1 连接 process 2 时就可以用网络地址127.0.0.1:2379。
  • 跨host的访问,比如process 3连接process 8,的时候就需要目标 host 的IP地址了。各个host的IP地址通常是网络中一个DHCP服务来统一分配的。通常DHCP服务运行在路由器上。
当我们启动一个分布式作业(job)的时候,我们会在很多机器上启动很多进程。首先我们得确保新启动的进程的port和那台机器上已有的进程的port不重复;然后我们还得让一个作业中所有进程互相知道对方的网络地址。
第一个问题好解决:当一个服务进程要监听一个port时,它可以调用操作系统的Listen功能调用,并且指定port。比如Go语言标准库对操作系统Listen功能调用的封装是net.Listen:
l, e := net.Listen("tcp", 0)
注意,当port是0时操作系统会分配一个空闲端口。这个端口加上本机IP构成的网络地址是:
l.Addr().String()
第二个问题也有解法:每个进程启动的时候,接受一个命令行参数,给这个进程指定一个名字;进程在调用Listen并且获取了port之后,把自己的名字和网络地址告诉机群上的一个公共服务,比如etcd,从而公示自己的网络地址,让其他进程可以连接自己。
但是一个最基本的问题并没有解决:在哪些机器上启动哪些进程?在一个小团队里,这个问题可以靠人和人协调:问问大家都在用那几台机器,从而找出相对空闲的几台,随后手工地把一个job里的多个进程分配到这几台机器上,利用ssh命令启动进程。
但是当人多了或者机器多了的时候,就得靠专门的软件来自动调度了。这时,一个进程可能被运行在机群中任何一台机器上。这就引发出部署(deploy) 的问题了。每个分布式程序有各自的依赖:有的需要Python 2.4,有的需要Python 2.6,有的需要Python 3.1。还有的需要特定版本的Java、Ruby、Lua、甚至GCC runtime library。如此缤纷,以至于没法在一个机群的所有机器上预先安装所有的依赖。这可怎么办?
一个直观的解法是把一个程序和它所有的依赖都预先安装在一个虚拟机镜像里。然后在需要启动程序的机器上运行一个虚拟机。

虚拟机

相对于上一节介绍的bare metal network,虚拟机是一个新概念:
  1. 真实节点(bare metal node),
  2. 虚拟机(virtual node),和
  3. 进程(process)
其中真实节点也被称为host,而虚拟机也被称为guest。
一个host上安装的虚拟机管理系统(比如VirtualBox和VMware)会运行一台或者多台guest,每个guest里运行一个guest操作系统(guest OS)和多个应用程序的进程。一个很自然的问题是:guest的IP地址如何分配?
大多数虚拟机管理系统提供两种IP地址分配方式:
  1. Bridged:虚拟机管理系统把IP地址分配的问题移交给host所在网络的IP地址分配机制(通常是host网络的路由器)。这样一来,每一个guest的IP地址和host一样,都是host网络的路由器分配的。启动一台guest,就像在网络里增加了一台机器一样。这样简单明了它的好处是管理很简单,因为host里的进程和guest里的进程都可以互相通信。坏处也很明显,一个物理网段里可以用的IP地址数量是有限的,所以bridged方式下,一个机群里能启动的guest的通常是有限的。这使得如果我们把每个VM当做一个机器来用,而不是只运行一个作业中的一个进程,那么IP地址还是够的。这是很多云服务的使用方式。
  2. NAT(network address translation): NAT模式下,每台host上的虚拟机管理系统虚拟一个“路由器”,负责给本host上的各个guest分配IP地址。因为guest IP和host IP不是同一个网段了,所以如果某个host里的进程要访问另一个host里某个guest里的进程的时候,要把目标guest的IP地址“翻译”成目标host的IP地址。这是NAT的名字的来历。知得注意的是:当各个host上都用NAT方式给guest分配IP的时候,在不同host上的多台guest的IP地址可能相同。
下图展示了NAT networking model:
  • 当一个进程要访问同一个guest里的另一个进程的时候,比如图中process 1要访问process 2的时候,可以用localhost网络地址。
  • 当跨guest但是不跨host的时候,用目标guest的guest IP定位目标进程。比如图中process 4访问process 2。
  • 当跨越host访问的时候就麻烦一些了:需要访问目标host的IP和host上的某个端口(图中所示8080),然后目标host上的虚拟机管理系统要能把host上8080端口映射到预先指定的VM 2中的80端口,这样对host-IP-2:8080的访问就被转发给host 2上的VM 2里的process 7了。这就叫端口转发(port foward)。

Docker

Docker的网络通信设计和虚拟机类似。Docker和虚拟机的主要区别在于:Docker不虚拟CPU和硬件,也不需要在虚拟硬件上运行guest OS。说实话,仅仅为了把程序及其依赖打个包方便部署,实在不需要虚拟机。所以相对于虚拟机群,Docker的路子里并没有增加新概念,只是把virtual node改成container了:
  1. 节点(node)
  2. 集装箱(container)
  3. 进程(process)
这篇文档描述了Docker的网络模型。下图更直观:
  • 和NAT模式下,每个host上的虚拟机管理系统虚拟一个路由器类似,Docker里有一个虚拟网桥docker0来负责给各个Docker container分配container IP。每一个container里有一个虚拟网卡eth0,和docker0连接。docker0分配的IP赋给了每个container的eth0。
  • 同一个container里的各个进程之间互相访问可以利用localhost IP。
  • 同一个host上不同container里的进程互访可以用同一个虚拟网桥下的container IP。比如图中process 4访问process 2时,process 2的网络地址是container-IP-1:2379。
  • 跨host的通信需要端口转发(port forward)。这和使用虚拟机的时候是类似的。比如图中process 3访问processs 7时,process 3连接的目标网络地址是host-IP-2:8080,其中 host 2上的8080端口被配置为转发到container 2里的80端口。

Kubernetes

用NAT模式的虚拟机群或者用Docker来组织分布式计算都有一个共同的麻烦:端口转发。当开发人员创建虚拟机镜像和Docker image的时候,是可以指定每个进程的局部(虚拟机内和container内)port的 。但是要支持跨host访问,还需要把这些局部端口映射到host端口。而映射到host上哪个端口,依赖当前恰好在host上执行的虚拟机或者container。
一个简单而且彻底的解决方法是要求机群上执行的所有程序都接受一个命令行参数,比如--port=,来指定进程监听的端口号;并且机群管理系统(比如Kubernetes或者Mesos或者YARN)在启动进程的时候,通过--port指定给进程一个端口号。但是很多常用程序,比如MySQL和nginx,都并不遵从这样的协议。很多进程会启动多个线程,同时监听多个端口。所以Kubernetes的设计得另寻出路。
Kubernetes的解决思路和Bridged模式的虚拟机群很像——用一个通用的IP地址分配服务,为运行在各个host上的container统一分配IP地址。这样运行在不同host上的containers之间通信,直接使用对方的container IP地址就可以了,而不需要考虑host IP。这实际上把Docker模式中的host IP和container IP这样两层IP地址变成了一层。
在Kubernetes的文档里阐述了一个叫Pod的概念,并且解释一个Pod里可以运行一个或者多个Docker containers。实际上,一个Pod就是一个Docker container。所谓在Pod里运行的多个containers,实际上是启动的时候加了--net=container:<pod-container>参数的containers,它们不会得到自己的IP地址,而是和pod container共享IP地址。这样一来,一个pod里的containers之间通信的时候可以用localhost地址,而跨越pod的通信用pod IP。 看上去Kubernetes的做法里相对于Docker的做法,多了一层Pod的概念。但是实际上每个container里约定俗成地只运行一个服务进程,所以还是三层概念:
  1. 节点(node)
  2. Pod
  3. container
图示如下
所以总体来说,Kubernetes的网络设计可以做到Pod之间透明的访问、Pod对外的透明访问:
  1. Pod可以不通过NAT就能访问集群中其他所有的容器
  2. 所有的node可以不通过NAT访问到Pod,反之亦然
  3. Pod自己看到自己的IP和其他的地方看到的IP是同一个
这篇文档列出了在一个机群里为每一个Pod分配一个唯一且可以路由的IP地址的几种方法。我们将着重考虑利用flannel的方法。并且在这里介绍。
关于Pod和Service之间的通信方式,将会在这里进行介绍。

Network virtualization & SDN

network virtualization 使用软件和硬件,将网络资源组成一个可以方便控制和直接操作的整体,将逻辑网络和底层的物理网络的配置解。通常包括虚拟化的L2-L3交换机,路由器,网卡,甚至L4-L7的防火墙,load banlancer。虚拟网络通常会将VLAN(VxLAN)网络或者VM或Containers连接起来,构成一个整体。
软件定义网络(Software Defined Network, SDN ) 由于传统的网络设备(交换机、路由器)的固件是由设备制造商锁定和控制,所以SDN希望将网络控制与物理网络拓扑分离,从而摆脱硬件对网络架构的限制。这样便可以像升级、安装软件一样对网络架构进行修改,满足企业对整个网站架构进行调整、扩容或升级。而底层的交换机、路由器等硬件则无需替换,节省大量的成本的同时,网络架构迭代周期将大大缩短。通过下面的一些链接来更深入的了解SDN:
不论是物理服务器,VM还是容器,对于大规模的集群组网,使用SDN的方式,可以实现动态组网,自动扩容,租户隔离等。可以参考一个常用的SDN的实现:Open vSwitch(简称OVS)。一个常见的OVS的网络拓扑如下:
原文地址:
转载时必须以链接形式注明原始出处及本声明。

Wednesday, May 24, 2017

OpenStack Magnum provides APIs and tenant isolation for container orchestration engines

1.Complete management of containers within OpenStack
  • Orchestrates the underlying host machines with Heat
  • Implements multi-tenancy of separate clusters through Keystone
  • Provides multi-host networking with Neutron
2.Supports several Container Orchestration Engines(COE)
  • Docker Swarm
  • Google Kubernetes
  • Apache Mesos
3.Allows direct access to native container APIs
  • Docker CLI clients can access hosts and containers
  • The Kubernetes client can also directly manage pods, services, etc.

Several OpenStack projects leverage containers to gain benefits

1. Nove
A Docker hypervisor driver for Nova Compute to treat containers and images as the same type of resources as virtual machines.
2. Heat
A plugin template for orchestrating Docker resources on top of OpenStack resources. Allows access to full Docker API.
3. Kolla
Containerizes the OpenStack control services themselves as microservices to simplify the oprational experience.
4.Murano
Provides an application catalog of containerized applicatoins that can be deployed to an OpenStack cloud.
5.Magnum
Provides an API to manage multi-tenant Containers-as-a-Service leveraging Hear, Nova, and Neutron.


OpenStack is above all an integration engine, bringing various technologies together through common APIs. Therefore, containers have naturally been plugged into serveral existing projects and will find their way into other areas as well.

Monday, May 22, 2017

Magento2 Setup new extension

1.upload to server magento directory
2.setup Magento2 extension:
php bin/magento setup:upgrade
3.deploy static files
php bin/magento setup: static-content:deploy

Saturday, May 13, 2017

How to Install Python 3.5.2 on Ubuntu

1.sudo apt-get install build-essential checkinstall
2.sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
3.cd /usr/src
4.sudo wget https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz
5.sudo tar xzf Python-3.5.2.tgz
6.cd Python-3.5.2
7.sudo ./configure
8.sudo make altinstall
9.python3.5 -V 
 

Ubuntu Add Startup Teamviewer program

Go to start up applications
Click “Add”
Now you will see a box called Add Startup Program box
Now In Name place Type Start Team viewer
In Command place type /usr/bin/teamviewer
And now press Add button
And press close button.

Docker and OpenStack Integration


Deploy OpenStack via Docker

       Docker contains OpenStack services, and implements like server nodes.

Deploy Docker in OpenStack

  1. Docker driver for Nova
  2. Docker Plugin for Heat
  3. Magnum

Friday, May 12, 2017

php bin/magento catalog:image:resize Unsupported image format

There are missing product images after you imported products into Magento.
You can even reimport product with images or just edit on admin panel.

Most used command line in Magento 2


  1. Install module and upgrade module: php bin/magento setup:upgrade
  2. Switch to the developer mode or production mode: php bin/magento deploy:mode:set developer (or production)
  3. Show current mode: php bin/magento deploy:mode:show
  4. Deploy static content: php bin/magento setup:static-content:deploy
  5. Di compile: php bin/magento setup:di:compile
  6. Reindex: php bin/magento indexer:reindex
  7. Flush magento cache: php bin/magento cache:flush

Magento 2 How to unlock reindex process

1. Find the indexes name
php bin/magento indexer:info
2.Check the indexes status
php bin/magento indexer:status
3.Reset all the indexes or reset specific indexes
php bin/magento indexer:reset
php bin/magento indexer:reset cataloginventory_stock
4. Reindex
php bin/magento indexer:reindex
5.Clean the cache
php bin/magento cache:clean

Thursday, May 11, 2017

Catalog Search indexer process unknown error: There is no such engine

Open magento root directory path and do following commands:
php -f bin/magento setup:upgrade
php -f bin/magento cache:flush
php -f bin/magento indexer:reindex

Saturday, May 6, 2017

Devstack Installation Error: Permission Denied

check the owner and permissions of the folder devstack, where you cloned the code, set it by command chown -R stack devstack and then also change permissions by chmod 770 devstack

How to Enable SSH in Ubuntu 16.04 LTS

1.sudo apt-get install openssh-server
2.sudo service ssh status

Thursday, May 4, 2017

Longest Palindromic Substring

Given a string s, find the longest palindromic substring in s. You may assume that the maximum length of s is 1000.
Example:
Input: "babad"

Output: "bab"

Note: "aba" is also a valid answer.
Example:
Input: "cbbd"

Output: "bb"
Thoughts:

1. Brute Force
 It is the possibility of exhausting all the substrings, and then step by step to determine whether it is a palindrome and update the results. Although its time complexity is high, but its space requirements are very low.

public class Solution {
    public String longestPalindrome(String s) {
        int maxLength = 0;
        int maxStart = 0;
        int len = s.length();
        for(int i = 0; i < len; i++){
            //i is the length of s
            for(int j = 0; j < len - i; j++){
                //j is the start position
                if(isPalindrome(s, i , j) && (i + 1) > maxLength){
                    maxLength = i + 1;
                    maxStart = j;
                }
            }
        }
        return s.substring(maxStart, maxStart + maxLength);
    }
    private boolean isPalindrome(String s, int i, int j){
        int left = j;
        int right = j + i;
        while(left < right){
            if(s.charAt(left) != s.charAt(right)){
                return false;
            }
            left++;
            right--;
        }
        return true;
    }
}
2. Dynamic Programming
According to the characteristics of the palindrome, a large palette after the proportion of the string must also be a palindrome, such as ABCCBA, that BCCB is also a palindrome. So we can according to the two characteristics of dynamic programming: the first big problem dismantling for small problems, the second repeated use of the previous calculation results, to answer this question. How can we divide the small problem, we can first all the shortest length of 1 sub-string calculated, according to the starting position from left to right, these must be palindrome. And then calculate all the length of 2 sub-string, and then according to the starting position from left to right. To the length of 3, we can use the results of the last calculation: If the center of the short string is not a palindrome, that long string is not, if the short string is palindrome, it depends on the long string Whether the two are the same. In this way, until the longest sub-string, we put the entire string set exhaustive, but due to the use of dynamic programming, so that the calculation time from O (N ^ 3) reduced to O (n ^ 2).

public class Solution {
    public String longestPalindrome(String s) {
        int maxLength = 0;
        int maxStart = 0;
        int len = s.length();
        boolean[][] dp = new boolean[len][len];
        for(int i = 0; i < len; i++){
            //i is the length of s
            for(int j = 0; j < len - i; j++){
                //j is the start position
                if(i == 0 || i == 1){
                    dp[j][j + i] = true;
                    //if start == end check center
                }else if(s.charAt(j + i) == s.charAt(j)){
                    dp[j][j + i] = dp[j + 1][j + i - 1];
                }else{
                    dp[j][j + i] = false;
                }
                if(dp[j][j + i] && i + 1 > maxLength){
                    maxLength = i + 1;
                    maxStart = j;
                }
            }
        }
        return s.substring(maxStart, maxStart + maxLength);
    }
 
}

Wednesday, May 3, 2017

DevStack installation to build private cloud public cloud openstack environment

DevStack is a series of extensible scripts used to quickly bring up a complete OpenStack environment based on the latest versions of everything from git master. It is used interactively as a development environment and as the basis for much of the OpenStack project’s functional testing.

./stack.sh: line 489: generate-subunit: command not found

Installed python dependencies:
sudo apt-get install python-pip
sudo pip install --upgrade pip
sudo pip install -U os-testr
Installed brctl dependency:
sudo apt-get install bridge-utils
Changed the configuration from git:
git config --global url."https://".insteadOf git://

line 232: /opt/stack/logs/error.log: No such file or directory

You can try and set this variable:
FORCE=yes ./stack.sh 

Majority Element -LeetCode

Given an array of size n, find the majority element. The majority element is the element that appears more than ⌊ n/2 ⌋ times.
You may assume that the array is non-empty and the majority element always exist in the array.

Magento 2 catalog image not showing, or product image show on backend but not frontend.



For individual product, try saving the product in admin without any change.
For all products, try running the following command (this will take some time to complete):
php bin/magento catalog:images:resize

Monday, May 1, 2017

Magento exception printing is disabled by default for security reasons – How to fix?

1.Clear Magento cache in /var/cache and refresh your page to see if it works!
2. Eenable exception printing in pub/errors/local.xml. You will need to rename the local.xml.sample to local.xml. Then you can refresh your page to see the printed error.