Regular Expression to minimum(non-greedy) matching

If we would like to match multiple duplicate content in the same line, we would like to use minimum match by using RE.

for example:

filename: text

pillar[‘abc’]
pillar[‘abc’]
pillar[‘abc’]}pillar[‘wx’]||pillar[‘yz’]
pillar[‘abc’]

1. matched expression python

import re

f=open('./text','r').read()
# non-greddy result:

print(re.findall('(pillar\[.*?\])',f))

# ourput: ["pillar['abc']", 
"pillar['abc']", 
"pillar['abc']", 
"pillar['wx']", 
"pillar['yz']", 
"pillar['abc']"]

# greedy match result:

print(re.findall('(pillar\[.*\])',f))

#output: ["pillar['abc']", 
"pillar['abc']", 
"pillar['abc']}pillar['wx']||pillar['yz']", 
"pillar['abc']"]

 

2. matched expression using shell

grep -o -P "pillar\['.*?'\]" text

# output: 
pillar['abc']
pillar['abc']
pillar['abc']
pillar['wx']
pillar['yz']
pillar['abc']

LDAP_ host-based limited with Pam_filter Not Working

Recently, I was trying to migrate our all of the stuffs of LDAP into MysqlDB. The password with type: md5crypt (not very clear why using this by default). When using PAM_FILTER modules for host limiting, however it was failed. So googled find this solution. This is very helpful for me. Thanks to the author for this. 🙂

Reference link: http://computingfunnyfacts.blogspot.com/2008/01/pamfilter-not-working.html

Here is the blog content of :system-auth

config of /etc/ldap.conf

It’s working!
:::::::::::::::::::::::::::::::::::::::
pam_filter not working
So here is the problem you want to limit your cluster to a special user group. You have everything LDAP managed and use pam_ldap for authentication. But when you edit the /etc/ldap.conf and set a pam_filter nothing happens. First of all the the syntax of pam_filter :
(|(gidNumber=1028)(gidNumber=1160))
Will not work
Literally only
pam_filter gidNumber=1028
Will work. This is the way they stupidly implemented it
else if (!strcasecmp (k, “pam_filter”))
{
CHECKPOINTER (result->filter = strdup (v));
}

where v is everything after the ‘ ‘

while (*v != ‘\0’ && *v != ‘ ‘ && *v != ‘\t’)
v++;

*(v++) = ‘\0’;

For those that know C
So you can give it one value max. Now you have to modify the /etc/pam.d/system-auth file. The default configuration is:
[root@lxb5477 ~]# cat /etc/pam.d/system-auth.back | grep ldap
auth sufficient /lib/security/$ISA/pam_ldap.so use_first_pass
account [default=bad success=ok user_unknown=ignore] /lib/security/$ISA/pam_ldap.so
password sufficient /lib/security/$ISA/pam_ldap.so use_authtok
session optional /lib/security/$ISA/pam_ldap.so

But of course a optional is not really enough. You, of course, want that if the user doesn’t fulfill your filter he should be chucked into nirvana. So change to:
[root@lxb5477 ~]# cat /etc/pam.d/system-auth | grep ldap
auth required /lib/security/$ISA/pam_ldap.so
account required /lib/security/$ISA/pam_ldap.so
password requried /lib/security/$ISA/pam_ldap.so use_authtok
session required /lib/security/$ISA/pam_ldap.so

Through this if ldap fails the login fails.

But be aware that in /etc/nsswitch.conf files is before ldap
passwd: files ldap
shadow: files ldap
group: files ldap

Setting up LDAP autentication with sudoer access

Building Environment:
==================================
OS: CentOS 5.4(64 bit)
==================================
Packages:openldap-servers-2.3.43-12.el5_7.10
openldap-devel-2.3.43-12.el5_7.10
openldap-2.3.43-12.el5_7.10
openldap-2.3.43-12.el5_7.10
openldap-clients-2.3.43-12.el5_7.10
openldap-devel-2.3.43-12.el5_7.10
===================================
sudo-1.6.9p17-5.el5
===================================
Optional: WEB UI
phpldap
===================================

#### Server settings ######
1. Install openldap-server

yum install openldap-servers openldap-devel openldap openldap-clients

2. Config sldap.conf
# set schemas
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
include /etc/openldap/schema/sudoer.schema
include /etc/openldap/schema/corba.schema
include /etc/openldap/schema/dyngroup.schema
include /etc/openldap/schema/misc.schema
include /etc/openldap/schema/openldap.schema
include /etc/openldap/schema/ppolicy.schema

# DB type & dc setting
database bdb
suffix “dc=happyelements,dc=net”
rootdn “cn=manager,dc=happyelements,dc=net”
rootpw secret # this can by generated by “slappasswd”
#
directory /usr/local/openldap/var/openldap-data

3. /etc/openldap/ldap.conf for searching by ldapsearch

URI ldap://127.0.0.1/
BASE dc=happyelements,dc=net
TLS_CACERTDIR /etc/openldap/cacerts

4. Copy sudo schema to /etc/schema
cp /usr/share/doc/sudo-1.6.9p17/schema.OpenLDAP /etc/openldap/schema/sudoer.schema

5. Set database dir:
mkdir -p /usr/local/openldap/var/openldap-data
chown ldap:ldap /usr/local/openldap/var/openldap-data
cp /etc/openldap/DB_CONFIG.example /usr/local/openldap/var/openldap-data/DB_CONFIG

6. Testing Servers
/etc/init.d/ldap start

7. Create entries related for LDAP Server:

#Paste following lines as file: happyelements.ldif
dn: dc=happyelements,dc=net
objectClass: dcObject
objectClass: organization
dc: happyelements
o: happyelements.net
description: happyelements.net

dn: cn=manager,dc=happyelements,dc=net
objectClass: organizationalRole
cn: manager

dn: ou=Group,dc=happyelements,dc=net
objectClass: organizationalUnit
ou: Group

dn: ou=People,dc=happyelements,dc=net
objectClass: organizationalUnit
ou: People

#Paste following lines into file sudoeraccess.ldif

objectClass: top
objectClass: organizationalUnit
ou: SUDOers

dn:cn=defaults,ou=SUDOers,dc=happyelements,dc=net
cn:defaults
sudoOption:ignore_dot
sudoOption:!mail_no_user
sudoOption:!root_sudo
sudoOption:log_host
sudoOption:logfile=/var/log/sudolog
sudoOption:!syslog
sudoOption:timestamp_timeout=10
objectClass:top
objectClass:sudoRole
description:DefaultsudoOption’s

dn:cn=Rule1,ou=SUDOers,dc=happyelements,dc=net
cn:Rule1
sudoOption:!authenticate
objectClass:top
objectClass:sudoRole
sudoHost:ALL
sudoCommand:ALL
sudoUser:ALL
description:AllowedwithoutpasswordforALLusers

Add entries into LDAP Server:
ldapadd -x -D “cn=manger,dc=happyelements,dc=net” -W -f happyelements.ldif
ldapadd -x -D “cn=manger,dc=happyelements,dc=net” -W -f sudoeraccess.ldif

ADD User entries:
# paste following lines into user_passswd.ldif
uid: root
cn: root
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$1$DpCxWpSc$E1Tsbg/CFnP1MZhvXqCdg1
shadowLastChange: 15117
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 0
gidNumber: 0
homeDirectory: /root
gecos: root

dn: uid=rma1,ou=People,dc=happyelements,dc=net
uid: rma1
cn: rma1
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$1$70S56nN0$rW/Sjk0rCrem4s3.emGun.
shadowLastChange: 15399
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 10402
gidNumber: 10402
homeDirectory: /home/rma1

dn: uid=rma2,ou=People,dc=happyelements,dc=net
uid: rma2
cn: rma2
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
userPassword: {crypt}$1$yfRr5GkD$lgF8Xu8cN92OMyR7tRlsK0
shadowLastChange: 15399
shadowMax: 99999
shadowWarning: 7
loginShell: /bin/bash
uidNumber: 10403
gidNumber: 10403
homeDirectory: /home/rma2

ADD group entries:
dn: cn=rma1,ou=Group,dc=happyelements,dc=net
objectClass: posixGroup
objectClass: top
cn: rma1
userPassword: {crypt}x
gidNumber: 10402

dn: cn=rma2,ou=Group,dc=happyelements,dc=net
objectClass: posixGroup
objectClass: top
cn: rma2
userPassword: {crypt}x
gidNumber: 10403

# Add entries into LDAP server

ldapadd -x -D “cn=manger,dc=happyelements,dc=net” -W -f user_passswd.ldif

# Add group entries

#paste following lines into ldif files with name: user_groups.ldif

#dn: cn=root,ou=Group,dc=happyelements,dc=net
#objectClass: posixGroup
#objectClass: top
#cn: root
#userPassword: {crypt}x
#gidNumber: 0

dn: cn=rma1,ou=Group,dc=happyelements,dc=net
objectClass: posixGroup
objectClass: top
cn: rma1
userPassword: {crypt}x
gidNumber: 10402

dn: cn=rma2,ou=Group,dc=happyelements,dc=net
objectClass: posixGroup
objectClass: top
cn: rma2
userPassword: {crypt}x
gidNumber: 10403

#ldapadd -x -D “cn=manger,dc=happyelements,dc=net” -W -f user_groups.ldif

### Client Settings ###
8. LDAP client settings

8.1 edit /etc/ldap.conf like this:

sudoers_base ou=SUDOers,dc=happyelements,dc=net
base dc=happyelements,dc=net
timelimit 120
bind_timelimit 120
idle_timelimit 3600
nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,radvd,tomcat,radiusd,news,mailman,nscd,gdm

nss_base_passwd ou=People,dc=happyelements,dc=net?one
nss_base_shadow ou=People,dc=happyelements,dc=net?one
nss_base_group ou=Group,dc=happyelements,dc=net?one

uri ldap://127.0.0.1
ssl no
tls_cacertdir /etc/openldap/cacerts
pam_password md5

8.2 edit /etc/nsswitch.conf

sudoers: ldap files

That’s all we need do. Enjoy.

APPINDEX 1:
Reference links
Official sudoer manual: http://www.sudo.ws/sudo/man/1.8.4/sudoers.ldap.man.html
Setting Up A Centralised Authentication Server With Sudo Access Using LDAP:
http://fci.wikia.com/wiki/Setting_Up_A_Centralised_Authentication_Server_With_Sudo_Access_Using_LDAP
posix migration tools: http://www.padl.com/download/MigrationTools.tgz

APPINDEX 2:
3 methods to enforce host-based authentication:
using pam_check_host_attr in /etc/ldap.conf
#pam_check_host_attr yes
using pam_filter authentication in /etc/ldap.conf
#pam_filter |(host=10.130.142.103) (host=\*)
using nss_base_

authentication in /etc/ldap.conf

Continue reading “Setting up LDAP autentication with sudoer access”

自动化运维之路[转]

Reference Link: http://www.dbanotes.net/web/web_operations_automatic.html

还是继续这个网站运维的话题吧。前面谈了知识管理与积累,这次谈一下运维过程中的自动化管理。

在进行这篇的扯淡之前,让我想起了《太平广记》里的一个《 板桥三娘子》的故事,姓赵的客商窥探到客栈老板娘三娘子在小箱子中取出小孩玩具大小的木头牛,木头人,喷口水,木头人、牛开始犁地耕作,撒一粒荞麦种子,木头小人种下,不一会儿,荞麦长成开花结实,木头人收割,乃至磨成面粉。然后三娘子把木头牛、人收入箱中,用得来的面粉做了数张面饼。多么好的一个自动化场景呀。

自动化的目的

自动化管理是网站规模化之后必须要面对的问题。为什么要自动化?肯定不是为了炫技,针对一个发展中的网站来说,自动化的主要目的还是为了节省维护成本,提升运维成熟度能力。另外一个经常被忽略的收益是能让运维工作更有趣味性一些,不那么无聊,不无聊的有益副作用是减少人为出错的可能。

自动化针对的范围大致可以分为安装自动化、部署自动化、软件发布自动化、升级自动化、监控自动化等几个方面。优化自动化? 别,这个稍微”高级”并且不靠谱了一点。

自动化要解决的问题是 N 次循环的过程,如果 N 不具备延续性,那么自动化未必有必要。比如某个过程可能只是短时间内需要临时进行几次,是否有必要将其自动化就有待于商榷。如果计划和开发自动化过程的成本高于非自动化成本就没必要了。

开发自动化过程

如果看过古龙的小说,他曾经描述过几个有趣的懒人,懒人造了一些木头人和机关来帮自己干一些不愿意做的事情。自动化多少就是”懒人”要做的事情,因为懒嘛,所以才会想办法节省时间和其他成本。一般来说,这个过程的开发者也是使用者,所以没必要一定要按照所谓的项目过程去走,但是开发者必须能够产出一份文档给同团队的伙伴(如果有的话)。

考虑到多数的网站运维可能都是在 Unix like 环境中的,而 Unix 的哲学思想之一就是”Write programs that do one thing and do it well”,每个过程只做一件事情就很关键,”功能单一的自动化模块”是有必要的,把不同的模块拼装起来再去完成更复杂的需求。

Unix 相比 Windows 来说,天生具备可自动化能力。如 Shell/BASH(自动化日常操作)、CronTab(自动化任务调度) 、Expect (自动化交互场景) 、rsync(数据远程同步)等 啊都是一些需要注意的技术内容。

优化自动化过程

自动化过程一般要有个生命周期,定期升级、优化也是有必要的。面对不同的应用场景应该逐渐改进自动化的可用性。

示例:自动部署 Linux

对于批量的 Linux 安装,RedHat 提供有 Kickstart Installations 自动安装解决方案,不过该方案相对比较繁琐,前不久推出的 Cobbler 是让人眼前一亮的好工具(参见 hutuworm 的介绍文章)。我一直怀疑 Cobbler 是中国人命名的项目,因为 PXE 发音为”pixie”(皮鞋),而 Cobbler 的中文意思是”补鞋匠”。

OS 安装完毕之后的软件安装、更新是个麻烦事。在一个 Linux 的环境中,SA 一定不要为软件相互依赖性浪费太多的时间。什么 YUM、APT、YAST 啊,能用就用上。别太迷信自己编译软件所能带来的优化收益,实际上犯错的几率更大。达到某个规模后,本地建立、维护一个软件资料库(repositories)也是有必要的。

Linux 软件安装进化之路:

手工预编译-->RPM-->APT 等工具

已经进化到更好的阶段了,没必要还走着老路在原地折腾。

其他参考:Flickr 运维曾经采用 System Image 来自动化 Linux 相关的的运维工作。或许也可以尝试一下。

在系统配置管理(别混淆到另一个配置管理上去)方面,其实 cfengine 就挺好用的。更多类似工具参考这个比较列表。

标准化,减少后续维护成本是节省人力资源的一大法门。

自动化的一些风险

必须要承认的是,自动化有的时候是容易带来一些风险的,比如”冲掉”原有配置文件信息,不恰当的自动化脚本给系统带来额外负载等,在运维过程中需要不断总结经验。(又落入俗套)

这方面值得推荐的一本书是《UNIX和Linux自动化管理》,借鉴一下其中的思路和方法。

对了,补充一下前面的《板桥三娘子》的故事发展,三娘子的面饼如果被客人吃下,则会变成驴…… 同样,自动化有的时候会把人陷进去的,运维人不要变成自动化的奴隶。

这个话题还需要继续下去么? 我再想想 …